2026-02-22T19:44:38.769 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-22T19:44:38.787 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T19:44:38.893 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974 branch: wip-yuri6-testing-2026-02-22-1629-squid description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-journaling} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '63974' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run the rbd_mirror.sh workunit to test the rbd-mirror daemon in snapshot mode name: yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri6-testing-2026-02-22-1629-squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2026-02-22-1629-squid sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9196 sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuri6-testing-2026-02-22-1629-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 targets: trial007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJMUlU4Cteug0GiCx5j2Uet2DqUnnKtQrIB7U1GQif+JQTZksVOpHfZDaz1UwK4wK9qYTa3nRdx4FZj5aIdfkyo= trial017.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLDW5vyj+GviZNpv6neyV3hYsSiL3/fKOpVwfikmufzMXNVE4sj5SIJFR4lUNFi/6pYMJ2OdRg0h3yjhc8eG8CM= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror.sh env: CEPH_ARGS: '' RBD_IMAGE_FEATURES: layering,exclusive-lock,journaling RBD_MIRROR_CONFIG_KEY: '1' RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-22_19:35:38 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-22T19:44:38.893 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa; will attempt to use it 2026-02-22T19:44:38.893 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks 2026-02-22T19:44:38.894 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-22T19:44:38.895 INFO:teuthology.task.internal:Checking packages... 2026-02-22T19:44:38.895 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f90f4eaebb28b619cf76e80cbbe997656da69579' 2026-02-22T19:44:38.895 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-22T19:44:38.895 INFO:teuthology.packaging:ref: None 2026-02-22T19:44:38.895 INFO:teuthology.packaging:tag: None 2026-02-22T19:44:38.895 INFO:teuthology.packaging:branch: wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T19:44:38.895 INFO:teuthology.packaging:sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:44:38.895 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T19:44:39.014 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:44:39.016 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-22T19:44:39.027 INFO:teuthology.task.internal:no buildpackages task found 2026-02-22T19:44:39.027 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-22T19:44:39.040 INFO:teuthology.task.internal:Saving configuration 2026-02-22T19:44:39.051 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-22T19:44:39.066 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-22T19:44:39.130 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial007.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 19:37:55.917641', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINQIBDhBcLcAE41UbIhNCngOQk3Y2n/zVWgRtfzg8KYH'} 2026-02-22T19:44:39.191 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial017.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 19:37:55.918249', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAypIU/tmTtX4Vd+vsn94/XiNjcwAXW8bcheRpufpD7r'} 2026-02-22T19:44:39.191 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-22T19:44:39.202 INFO:teuthology.task.internal:roles: ubuntu@trial007.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2026-02-22T19:44:39.202 INFO:teuthology.task.internal:roles: ubuntu@trial017.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2026-02-22T19:44:39.202 INFO:teuthology.run_tasks:Running task console_log... 2026-02-22T19:44:39.350 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7060f1276de0>, signals=[15]) 2026-02-22T19:44:39.350 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-22T19:44:39.361 INFO:teuthology.task.internal:Opening connections... 2026-02-22T19:44:39.361 DEBUG:teuthology.task.internal:connecting to ubuntu@trial007.front.sepia.ceph.com 2026-02-22T19:44:39.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T19:44:39.426 DEBUG:teuthology.task.internal:connecting to ubuntu@trial017.front.sepia.ceph.com 2026-02-22T19:44:39.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T19:44:39.493 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-22T19:44:39.505 DEBUG:teuthology.orchestra.run.trial007:> uname -m 2026-02-22T19:44:39.509 INFO:teuthology.orchestra.run.trial007.stdout:x86_64 2026-02-22T19:44:39.509 DEBUG:teuthology.orchestra.run.trial007:> cat /etc/os-release 2026-02-22T19:44:39.553 INFO:teuthology.orchestra.run.trial007.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-22T19:44:39.554 INFO:teuthology.orchestra.run.trial007.stdout:NAME="Ubuntu" 2026-02-22T19:44:39.554 INFO:teuthology.orchestra.run.trial007.stdout:VERSION_ID="22.04" 2026-02-22T19:44:39.554 INFO:teuthology.orchestra.run.trial007.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-22T19:44:39.554 INFO:teuthology.orchestra.run.trial007.stdout:VERSION_CODENAME=jammy 2026-02-22T19:44:39.554 INFO:teuthology.orchestra.run.trial007.stdout:ID=ubuntu 2026-02-22T19:44:39.554 INFO:teuthology.orchestra.run.trial007.stdout:ID_LIKE=debian 2026-02-22T19:44:39.554 INFO:teuthology.orchestra.run.trial007.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-22T19:44:39.554 INFO:teuthology.orchestra.run.trial007.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-22T19:44:39.554 INFO:teuthology.orchestra.run.trial007.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-22T19:44:39.554 INFO:teuthology.orchestra.run.trial007.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-22T19:44:39.554 INFO:teuthology.orchestra.run.trial007.stdout:UBUNTU_CODENAME=jammy 2026-02-22T19:44:39.554 INFO:teuthology.lock.ops:Updating trial007.front.sepia.ceph.com on lock server 2026-02-22T19:44:39.613 DEBUG:teuthology.orchestra.run.trial017:> uname -m 2026-02-22T19:44:39.617 INFO:teuthology.orchestra.run.trial017.stdout:x86_64 2026-02-22T19:44:39.617 DEBUG:teuthology.orchestra.run.trial017:> cat /etc/os-release 2026-02-22T19:44:39.662 INFO:teuthology.orchestra.run.trial017.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-22T19:44:39.662 INFO:teuthology.orchestra.run.trial017.stdout:NAME="Ubuntu" 2026-02-22T19:44:39.662 INFO:teuthology.orchestra.run.trial017.stdout:VERSION_ID="22.04" 2026-02-22T19:44:39.662 INFO:teuthology.orchestra.run.trial017.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-22T19:44:39.662 INFO:teuthology.orchestra.run.trial017.stdout:VERSION_CODENAME=jammy 2026-02-22T19:44:39.662 INFO:teuthology.orchestra.run.trial017.stdout:ID=ubuntu 2026-02-22T19:44:39.662 INFO:teuthology.orchestra.run.trial017.stdout:ID_LIKE=debian 2026-02-22T19:44:39.662 INFO:teuthology.orchestra.run.trial017.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-22T19:44:39.663 INFO:teuthology.orchestra.run.trial017.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-22T19:44:39.663 INFO:teuthology.orchestra.run.trial017.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-22T19:44:39.663 INFO:teuthology.orchestra.run.trial017.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-22T19:44:39.663 INFO:teuthology.orchestra.run.trial017.stdout:UBUNTU_CODENAME=jammy 2026-02-22T19:44:39.663 INFO:teuthology.lock.ops:Updating trial017.front.sepia.ceph.com on lock server 2026-02-22T19:44:39.749 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-22T19:44:39.764 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-22T19:44:39.775 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-22T19:44:39.775 DEBUG:teuthology.orchestra.run.trial007:> test '!' -e /home/ubuntu/cephtest 2026-02-22T19:44:39.776 DEBUG:teuthology.orchestra.run.trial017:> test '!' -e /home/ubuntu/cephtest 2026-02-22T19:44:39.779 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-22T19:44:39.792 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-22T19:44:39.793 DEBUG:teuthology.orchestra.run.trial007:> test -z $(ls -A /var/lib/ceph) 2026-02-22T19:44:39.823 DEBUG:teuthology.orchestra.run.trial017:> test -z $(ls -A /var/lib/ceph) 2026-02-22T19:44:39.826 INFO:teuthology.orchestra.run.trial007.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-22T19:44:39.827 INFO:teuthology.orchestra.run.trial017.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-22T19:44:39.828 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-22T19:44:39.976 INFO:teuthology.run_tasks:Running task kernel... 2026-02-22T19:44:39.996 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-22T19:44:39.997 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-22T19:44:39.997 DEBUG:teuthology.orchestra.run.trial007:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T19:44:39.997 DEBUG:teuthology.orchestra.run.trial017:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T19:44:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:44:40.000 DEBUG:teuthology.orchestra.run.trial017:> uname -r 2026-02-22T19:44:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:44:40.001 DEBUG:teuthology.orchestra.run.trial007:> uname -r 2026-02-22T19:44:40.046 INFO:teuthology.orchestra.run.trial007.stdout:5.15.0-170-generic 2026-02-22T19:44:40.046 INFO:teuthology.task.kernel:Running kernel on trial007: 5.15.0-170-generic 2026-02-22T19:44:40.046 DEBUG:teuthology.orchestra.run.trial007:> sudo apt-get clean 2026-02-22T19:44:40.050 INFO:teuthology.orchestra.run.trial017.stdout:5.15.0-170-generic 2026-02-22T19:44:40.050 INFO:teuthology.task.kernel:Running kernel on trial017: 5.15.0-170-generic 2026-02-22T19:44:40.050 DEBUG:teuthology.orchestra.run.trial017:> sudo apt-get clean 2026-02-22T19:44:40.105 DEBUG:teuthology.orchestra.run.trial007:> sudo apt-get update 2026-02-22T19:44:40.108 DEBUG:teuthology.orchestra.run.trial017:> sudo apt-get update 2026-02-22T19:44:40.294 INFO:teuthology.orchestra.run.trial017.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T19:44:40.295 INFO:teuthology.orchestra.run.trial007.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T19:44:40.302 INFO:teuthology.orchestra.run.trial007.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-22T19:44:40.303 INFO:teuthology.orchestra.run.trial017.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-22T19:44:40.342 INFO:teuthology.orchestra.run.trial007.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-22T19:44:40.344 INFO:teuthology.orchestra.run.trial017.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-22T19:44:40.361 INFO:teuthology.orchestra.run.trial007.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-22T19:44:40.361 INFO:teuthology.orchestra.run.trial017.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-22T19:44:40.432 INFO:teuthology.orchestra.run.trial017.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-22T19:44:40.439 INFO:teuthology.orchestra.run.trial007.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-22T19:44:40.501 INFO:teuthology.orchestra.run.trial017.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-22T19:44:40.504 INFO:teuthology.orchestra.run.trial017.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-22T19:44:40.504 INFO:teuthology.orchestra.run.trial017.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-22T19:44:40.508 INFO:teuthology.orchestra.run.trial007.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-22T19:44:40.511 INFO:teuthology.orchestra.run.trial007.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-22T19:44:40.512 INFO:teuthology.orchestra.run.trial017.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-22T19:44:40.512 INFO:teuthology.orchestra.run.trial007.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-22T19:44:40.514 INFO:teuthology.orchestra.run.trial017.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-22T19:44:40.514 INFO:teuthology.orchestra.run.trial017.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-22T19:44:40.519 INFO:teuthology.orchestra.run.trial007.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-22T19:44:40.521 INFO:teuthology.orchestra.run.trial007.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-22T19:44:40.522 INFO:teuthology.orchestra.run.trial007.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-22T19:44:40.549 INFO:teuthology.orchestra.run.trial017.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-22T19:44:40.550 INFO:teuthology.orchestra.run.trial007.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-22T19:44:40.553 INFO:teuthology.orchestra.run.trial007.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-22T19:44:40.555 INFO:teuthology.orchestra.run.trial017.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-22T19:44:40.573 INFO:teuthology.orchestra.run.trial017.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-22T19:44:40.575 INFO:teuthology.orchestra.run.trial017.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-22T19:44:40.576 INFO:teuthology.orchestra.run.trial017.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-22T19:44:40.577 INFO:teuthology.orchestra.run.trial007.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-22T19:44:40.578 INFO:teuthology.orchestra.run.trial007.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-22T19:44:40.579 INFO:teuthology.orchestra.run.trial007.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-22T19:44:40.582 INFO:teuthology.orchestra.run.trial007.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-22T19:44:40.582 INFO:teuthology.orchestra.run.trial007.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-22T19:44:40.583 INFO:teuthology.orchestra.run.trial017.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-22T19:44:40.583 INFO:teuthology.orchestra.run.trial017.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-22T19:44:40.613 INFO:teuthology.orchestra.run.trial007.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-22T19:44:40.623 INFO:teuthology.orchestra.run.trial017.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-22T19:44:44.413 INFO:teuthology.orchestra.run.trial007.stdout:Fetched 22.6 MB in 2s (13.6 MB/s) 2026-02-22T19:44:44.424 INFO:teuthology.orchestra.run.trial017.stdout:Fetched 22.6 MB in 2s (13.7 MB/s) 2026-02-22T19:44:44.939 INFO:teuthology.orchestra.run.trial017.stdout:Reading package lists... 2026-02-22T19:44:44.946 INFO:teuthology.orchestra.run.trial007.stdout:Reading package lists... 2026-02-22T19:44:44.952 DEBUG:teuthology.orchestra.run.trial017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-22T19:44:44.960 DEBUG:teuthology.orchestra.run.trial007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-22T19:44:44.984 INFO:teuthology.orchestra.run.trial017.stdout:Reading package lists... 2026-02-22T19:44:44.992 INFO:teuthology.orchestra.run.trial007.stdout:Reading package lists... 2026-02-22T19:44:45.093 INFO:teuthology.orchestra.run.trial017.stdout:Building dependency tree... 2026-02-22T19:44:45.094 INFO:teuthology.orchestra.run.trial017.stdout:Reading state information... 2026-02-22T19:44:45.099 INFO:teuthology.orchestra.run.trial007.stdout:Building dependency tree... 2026-02-22T19:44:45.100 INFO:teuthology.orchestra.run.trial007.stdout:Reading state information... 2026-02-22T19:44:45.179 INFO:teuthology.orchestra.run.trial017.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-22T19:44:45.179 INFO:teuthology.orchestra.run.trial017.stdout:linux-image-generic set to manually installed. 2026-02-22T19:44:45.179 INFO:teuthology.orchestra.run.trial017.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T19:44:45.179 INFO:teuthology.orchestra.run.trial017.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T19:44:45.180 INFO:teuthology.orchestra.run.trial017.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T19:44:45.180 INFO:teuthology.orchestra.run.trial017.stdout: sg3-utils-udev 2026-02-22T19:44:45.180 INFO:teuthology.orchestra.run.trial017.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T19:44:45.184 INFO:teuthology.orchestra.run.trial007.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-22T19:44:45.185 INFO:teuthology.orchestra.run.trial007.stdout:linux-image-generic set to manually installed. 2026-02-22T19:44:45.185 INFO:teuthology.orchestra.run.trial007.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T19:44:45.185 INFO:teuthology.orchestra.run.trial007.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T19:44:45.185 INFO:teuthology.orchestra.run.trial007.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T19:44:45.185 INFO:teuthology.orchestra.run.trial007.stdout: sg3-utils-udev 2026-02-22T19:44:45.185 INFO:teuthology.orchestra.run.trial007.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T19:44:45.199 INFO:teuthology.orchestra.run.trial017.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T19:44:45.200 DEBUG:teuthology.orchestra.run.trial017:> dpkg -s linux-image-generic 2026-02-22T19:44:45.204 INFO:teuthology.orchestra.run.trial007.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T19:44:45.205 DEBUG:teuthology.orchestra.run.trial007:> dpkg -s linux-image-generic 2026-02-22T19:44:45.252 INFO:teuthology.orchestra.run.trial017.stdout:Package: linux-image-generic 2026-02-22T19:44:45.252 INFO:teuthology.orchestra.run.trial017.stdout:Status: install ok installed 2026-02-22T19:44:45.252 INFO:teuthology.orchestra.run.trial017.stdout:Priority: optional 2026-02-22T19:44:45.252 INFO:teuthology.orchestra.run.trial017.stdout:Section: kernel 2026-02-22T19:44:45.252 INFO:teuthology.orchestra.run.trial017.stdout:Installed-Size: 21 2026-02-22T19:44:45.252 INFO:teuthology.orchestra.run.trial017.stdout:Maintainer: Ubuntu Kernel Team 2026-02-22T19:44:45.252 INFO:teuthology.orchestra.run.trial017.stdout:Architecture: amd64 2026-02-22T19:44:45.252 INFO:teuthology.orchestra.run.trial017.stdout:Source: linux-meta 2026-02-22T19:44:45.252 INFO:teuthology.orchestra.run.trial017.stdout:Version: 5.15.0.170.159 2026-02-22T19:44:45.252 INFO:teuthology.orchestra.run.trial017.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-22T19:44:45.252 INFO:teuthology.orchestra.run.trial017.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-22T19:44:45.253 INFO:teuthology.orchestra.run.trial017.stdout:Recommends: thermald 2026-02-22T19:44:45.253 INFO:teuthology.orchestra.run.trial017.stdout:Description: Generic Linux kernel image 2026-02-22T19:44:45.253 INFO:teuthology.orchestra.run.trial017.stdout: This package will always depend on the latest generic kernel image 2026-02-22T19:44:45.253 INFO:teuthology.orchestra.run.trial017.stdout: available. 2026-02-22T19:44:45.253 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T19:44:45.253 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2026-02-22T19:44:45.253 DEBUG:teuthology.orchestra.run.trial017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T19:44:45.255 INFO:teuthology.orchestra.run.trial007.stdout:Package: linux-image-generic 2026-02-22T19:44:45.255 INFO:teuthology.orchestra.run.trial007.stdout:Status: install ok installed 2026-02-22T19:44:45.255 INFO:teuthology.orchestra.run.trial007.stdout:Priority: optional 2026-02-22T19:44:45.255 INFO:teuthology.orchestra.run.trial007.stdout:Section: kernel 2026-02-22T19:44:45.255 INFO:teuthology.orchestra.run.trial007.stdout:Installed-Size: 21 2026-02-22T19:44:45.256 INFO:teuthology.orchestra.run.trial007.stdout:Maintainer: Ubuntu Kernel Team 2026-02-22T19:44:45.256 INFO:teuthology.orchestra.run.trial007.stdout:Architecture: amd64 2026-02-22T19:44:45.256 INFO:teuthology.orchestra.run.trial007.stdout:Source: linux-meta 2026-02-22T19:44:45.256 INFO:teuthology.orchestra.run.trial007.stdout:Version: 5.15.0.170.159 2026-02-22T19:44:45.256 INFO:teuthology.orchestra.run.trial007.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-22T19:44:45.256 INFO:teuthology.orchestra.run.trial007.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-22T19:44:45.256 INFO:teuthology.orchestra.run.trial007.stdout:Recommends: thermald 2026-02-22T19:44:45.256 INFO:teuthology.orchestra.run.trial007.stdout:Description: Generic Linux kernel image 2026-02-22T19:44:45.256 INFO:teuthology.orchestra.run.trial007.stdout: This package will always depend on the latest generic kernel image 2026-02-22T19:44:45.256 INFO:teuthology.orchestra.run.trial007.stdout: available. 2026-02-22T19:44:45.256 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T19:44:45.256 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2026-02-22T19:44:45.257 DEBUG:teuthology.orchestra.run.trial007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T19:44:45.303 INFO:teuthology.orchestra.run.trial017.stdout:ttyS1 2026-02-22T19:44:45.306 INFO:teuthology.orchestra.run.trial007.stdout:ttyS1 2026-02-22T19:44:45.320 DEBUG:teuthology.parallel:result is None 2026-02-22T19:44:45.332 DEBUG:teuthology.parallel:result is None 2026-02-22T19:44:45.332 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-22T19:44:45.341 INFO:teuthology.task.internal:Creating test directory... 2026-02-22T19:44:45.341 DEBUG:teuthology.orchestra.run.trial007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T19:44:45.375 DEBUG:teuthology.orchestra.run.trial017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T19:44:45.379 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-22T19:44:45.391 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-22T19:44:45.402 INFO:teuthology.task.internal:Creating archive directory... 2026-02-22T19:44:45.402 DEBUG:teuthology.orchestra.run.trial007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T19:44:45.419 DEBUG:teuthology.orchestra.run.trial017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T19:44:45.429 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-22T19:44:45.438 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-22T19:44:45.438 DEBUG:teuthology.orchestra.run.trial007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T19:44:45.463 DEBUG:teuthology.orchestra.run.trial017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T19:44:45.471 INFO:teuthology.orchestra.run.trial007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T19:44:45.476 INFO:teuthology.orchestra.run.trial017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T19:44:45.477 INFO:teuthology.orchestra.run.trial007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T19:44:45.482 INFO:teuthology.orchestra.run.trial017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T19:44:45.483 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-22T19:44:45.493 INFO:teuthology.task.internal:Configuring sudo... 2026-02-22T19:44:45.493 DEBUG:teuthology.orchestra.run.trial007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T19:44:45.519 DEBUG:teuthology.orchestra.run.trial017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T19:44:45.535 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-22T19:44:45.545 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-22T19:44:45.545 DEBUG:teuthology.orchestra.run.trial007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T19:44:45.567 DEBUG:teuthology.orchestra.run.trial017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T19:44:45.578 DEBUG:teuthology.orchestra.run.trial007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T19:44:45.613 DEBUG:teuthology.orchestra.run.trial007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T19:44:45.657 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:44:45.657 DEBUG:teuthology.orchestra.run.trial007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T19:44:45.707 DEBUG:teuthology.orchestra.run.trial017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T19:44:45.711 DEBUG:teuthology.orchestra.run.trial017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T19:44:45.758 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:44:45.758 DEBUG:teuthology.orchestra.run.trial017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T19:44:45.808 DEBUG:teuthology.orchestra.run.trial007:> sudo service rsyslog restart 2026-02-22T19:44:45.809 DEBUG:teuthology.orchestra.run.trial017:> sudo service rsyslog restart 2026-02-22T19:44:45.868 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-22T19:44:45.874 INFO:teuthology.task.internal:Starting timer... 2026-02-22T19:44:45.874 INFO:teuthology.run_tasks:Running task pcp... 2026-02-22T19:44:45.890 INFO:teuthology.run_tasks:Running task selinux... 2026-02-22T19:44:45.901 DEBUG:teuthology.task.selinux:Excluding trial007: OS 'ubuntu' does not support SELinux 2026-02-22T19:44:45.901 DEBUG:teuthology.task.selinux:Excluding trial017: OS 'ubuntu' does not support SELinux 2026-02-22T19:44:45.902 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-22T19:44:45.902 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-22T19:44:45.902 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-22T19:44:45.902 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-22T19:44:45.911 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-22T19:44:45.917 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-22T19:44:45.918 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial007.front.sepia.ceph.com,trial017.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-22T19:46:51.824 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial007.front.sepia.ceph.com'), Remote(name='ubuntu@trial017.front.sepia.ceph.com')] 2026-02-22T19:46:51.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial007.front.sepia.ceph.com' 2026-02-22T19:46:51.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T19:46:51.892 DEBUG:teuthology.orchestra.run.trial007:> true 2026-02-22T19:46:52.136 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial007.front.sepia.ceph.com' 2026-02-22T19:46:52.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial017.front.sepia.ceph.com' 2026-02-22T19:46:52.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T19:46:52.204 DEBUG:teuthology.orchestra.run.trial017:> true 2026-02-22T19:46:52.447 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial017.front.sepia.ceph.com' 2026-02-22T19:46:52.447 INFO:teuthology.run_tasks:Running task clock... 2026-02-22T19:46:52.462 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-22T19:46:52.462 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T19:46:52.462 DEBUG:teuthology.orchestra.run.trial007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T19:46:52.463 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T19:46:52.463 DEBUG:teuthology.orchestra.run.trial017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T19:46:52.480 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-22T19:46:52.480 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: Command line: ntpd -gq 2026-02-22T19:46:52.480 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: ---------------------------------------------------- 2026-02-22T19:46:52.480 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: ntp-4 is maintained by Network Time Foundation, 2026-02-22T19:46:52.480 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-22T19:46:52.480 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: corporation. Support and training for ntp-4 are 2026-02-22T19:46:52.480 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: available at https://www.nwtime.org/support 2026-02-22T19:46:52.480 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: ---------------------------------------------------- 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: proto: precision = 0.030 usec (-25) 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: basedate set to 2022-02-04 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: gps base set to 2022-02-06 (week 2196) 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: restrict ::: KOD does nothing without LIMITED. 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: Listen and drop on 0 v6wildcard [::]:123 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: Listen normally on 2 lo 127.0.0.1:123 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: Listen normally on 3 enp1s0f0 10.20.193.7:123 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: Listen normally on 4 lo [::1]:123 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:5e8a%2]:123 2026-02-22T19:46:52.481 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:51 ntpd[9434]: Listening on routing socket on fd #22 for interface updates 2026-02-22T19:46:52.510 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-22T19:46:52.510 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: Command line: ntpd -gq 2026-02-22T19:46:52.510 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: ---------------------------------------------------- 2026-02-22T19:46:52.510 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: ntp-4 is maintained by Network Time Foundation, 2026-02-22T19:46:52.510 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-22T19:46:52.510 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: corporation. Support and training for ntp-4 are 2026-02-22T19:46:52.510 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: available at https://www.nwtime.org/support 2026-02-22T19:46:52.510 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: ---------------------------------------------------- 2026-02-22T19:46:52.510 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: proto: precision = 0.020 usec (-25) 2026-02-22T19:46:52.510 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: basedate set to 2022-02-04 2026-02-22T19:46:52.510 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: gps base set to 2022-02-06 (week 2196) 2026-02-22T19:46:52.511 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T19:46:52.511 INFO:teuthology.orchestra.run.trial017.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T19:46:52.511 INFO:teuthology.orchestra.run.trial017.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-22T19:46:52.511 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: restrict ::: KOD does nothing without LIMITED. 2026-02-22T19:46:52.511 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: Listen and drop on 0 v6wildcard [::]:123 2026-02-22T19:46:52.511 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-22T19:46:52.511 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: Listen normally on 2 lo 127.0.0.1:123 2026-02-22T19:46:52.511 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: Listen normally on 3 enp1s0f0 10.20.193.17:123 2026-02-22T19:46:52.511 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: Listen normally on 4 lo [::1]:123 2026-02-22T19:46:52.511 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:647c%2]:123 2026-02-22T19:46:52.511 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:52 ntpd[9426]: Listening on routing socket on fd #22 for interface updates 2026-02-22T19:46:53.480 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:52 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:53.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:52 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:53.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:52 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:53.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:52 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:53.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:52 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:53.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:52 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:53.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:52 ntpd[9434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:53.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:52 ntpd[9434]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:53.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:53.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:53.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:53.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:53.550 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:53 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:53.550 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:53 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:53.550 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:53 ntpd[9434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:53.550 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:53 ntpd[9434]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:53.575 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:53.575 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:53.575 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:53.575 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:53.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:53.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:53.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:53.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:53 ntpd[9426]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:55.480 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:54 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:55.480 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:54 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:55.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:54 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:55.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:54 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:55.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:54 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:55.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:54 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:55.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:54 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:55.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:54 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:55.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:54 ntpd[9434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:55.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:54 ntpd[9434]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:55.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:55.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:55.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:55.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:55.551 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:55 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:55.551 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:55 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:55.551 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:55 ntpd[9434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:55.551 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:55 ntpd[9434]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:55.563 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:55 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:55.563 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:55 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:55.563 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:55 ntpd[9434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:55.563 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:55 ntpd[9434]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:55.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:55.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:55.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:55.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:55.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:55.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:55.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:55.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:55 ntpd[9426]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:56.480 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:55 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:56.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:55 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:56.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:56.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:56.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:56.511 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:56.544 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:56.545 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:56.545 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:56.545 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:56.551 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:56.551 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:56.551 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:56.551 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:56 ntpd[9426]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:57.480 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:56 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:56 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:56 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:56 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:56 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:56 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:56 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:56 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:56 ntpd[9434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:57.483 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:56 ntpd[9434]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:57.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.511 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.511 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.511 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.553 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:57 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.553 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:57 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.553 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:57 ntpd[9434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:57.553 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:57 ntpd[9434]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:57.563 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:57 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.563 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:57 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.563 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:57 ntpd[9434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:57.563 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:57 ntpd[9434]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:57.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:57.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:57.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:57.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:57.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:57.576 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:57 ntpd[9426]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:58.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:58.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:58.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:58.510 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:58.544 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:58.544 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:58.544 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:58.545 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:58.551 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:58.551 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:58.551 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:58.551 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:58 ntpd[9426]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:59.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:58 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:58 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:58 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.481 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:58 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.482 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:58 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.482 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:58 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.484 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:58 ntpd[9434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.484 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:58 ntpd[9434]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.484 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:58 ntpd[9434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:59.484 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:58 ntpd[9434]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:59.485 INFO:teuthology.orchestra.run.trial007.stdout:22 Feb 19:46:59 ntpd[9434]: ntpd: time set +0.502977 s 2026-02-22T19:46:59.485 INFO:teuthology.orchestra.run.trial007.stdout:ntpd: time set +0.502977s 2026-02-22T19:46:59.485 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:59 ntpd[9434]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-22T19:46:59.485 INFO:teuthology.orchestra.run.trial007.stderr:22 Feb 19:46:59 ntpd[9434]: can't open /var/log/ntpstats/loopstats.20260222: Permission denied 2026-02-22T19:46:59.511 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:59 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.511 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:59 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.511 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:59 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.511 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:59 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.543 INFO:teuthology.orchestra.run.trial007.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T19:46:59.543 INFO:teuthology.orchestra.run.trial007.stdout:============================================================================== 2026-02-22T19:46:59.543 INFO:teuthology.orchestra.run.trial007.stdout: 172-104-28-175. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:46:59.543 INFO:teuthology.orchestra.run.trial007.stdout: 144.202.41.38 ( .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:46:59.543 INFO:teuthology.orchestra.run.trial007.stdout: jane.qotw.net .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:46:59.543 INFO:teuthology.orchestra.run.trial007.stdout: ntp1.lax.us.hoj .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:46:59.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:59 ntpd[9426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T19:46:59.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:59 ntpd[9426]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T19:46:59.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:59 ntpd[9426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T19:46:59.574 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:59 ntpd[9426]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T19:46:59.574 INFO:teuthology.orchestra.run.trial017.stdout:22 Feb 19:46:59 ntpd[9426]: ntpd: time set +0.227727 s 2026-02-22T19:46:59.574 INFO:teuthology.orchestra.run.trial017.stdout:ntpd: time set +0.227727s 2026-02-22T19:46:59.575 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:59 ntpd[9426]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-22T19:46:59.575 INFO:teuthology.orchestra.run.trial017.stderr:22 Feb 19:46:59 ntpd[9426]: can't open /var/log/ntpstats/loopstats.20260222: Permission denied 2026-02-22T19:46:59.600 INFO:teuthology.orchestra.run.trial017.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T19:46:59.600 INFO:teuthology.orchestra.run.trial017.stdout:============================================================================== 2026-02-22T19:46:59.600 INFO:teuthology.orchestra.run.trial017.stdout: x.ns.gin.ntt.ne .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:46:59.600 INFO:teuthology.orchestra.run.trial017.stdout: as393746.mci.tr .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:46:59.600 INFO:teuthology.orchestra.run.trial017.stdout: 138.89.14.60 (p .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:46:59.600 INFO:teuthology.orchestra.run.trial017.stdout: 73.185.182.209 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T19:46:59.601 INFO:teuthology.run_tasks:Running task install... 2026-02-22T19:46:59.612 DEBUG:teuthology.task.install:project ceph 2026-02-22T19:46:59.612 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}} 2026-02-22T19:46:59.612 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'} 2026-02-22T19:46:59.613 INFO:teuthology.task.install:Using flavor: default 2026-02-22T19:46:59.617 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-22T19:46:59.617 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2026-02-22T19:46:59.617 DEBUG:teuthology.orchestra.run.trial007:> sudo apt-key list | grep Ceph 2026-02-22T19:46:59.618 DEBUG:teuthology.orchestra.run.trial017:> sudo apt-key list | grep Ceph 2026-02-22T19:46:59.661 INFO:teuthology.orchestra.run.trial007.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-22T19:46:59.689 INFO:teuthology.orchestra.run.trial007.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-22T19:46:59.689 INFO:teuthology.orchestra.run.trial007.stdout:uid [ unknown] Ceph.com (release key) 2026-02-22T19:46:59.691 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2026-02-22T19:46:59.692 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:46:59.697 INFO:teuthology.orchestra.run.trial017.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-22T19:46:59.797 INFO:teuthology.orchestra.run.trial017.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-22T19:46:59.797 INFO:teuthology.orchestra.run.trial017.stdout:uid [ unknown] Ceph.com (release key) 2026-02-22T19:46:59.798 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2026-02-22T19:46:59.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:46:59.887 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default/ 2026-02-22T19:46:59.887 INFO:teuthology.task.install.deb:Package version is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:46:59.913 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default/ 2026-02-22T19:46:59.913 INFO:teuthology.task.install.deb:Package version is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:47:00.243 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:47:00.243 DEBUG:teuthology.orchestra.run.trial007:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-22T19:47:00.252 DEBUG:teuthology.orchestra.run.trial007:> sudo apt-get update 2026-02-22T19:47:00.293 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:47:00.293 DEBUG:teuthology.orchestra.run.trial017:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-22T19:47:00.303 DEBUG:teuthology.orchestra.run.trial017:> sudo apt-get update 2026-02-22T19:47:00.484 INFO:teuthology.orchestra.run.trial017.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T19:47:00.488 INFO:teuthology.orchestra.run.trial017.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-22T19:47:00.497 INFO:teuthology.orchestra.run.trial017.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-22T19:47:00.507 INFO:teuthology.orchestra.run.trial017.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-22T19:47:00.663 INFO:teuthology.orchestra.run.trial017.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy InRelease 2026-02-22T19:47:00.731 INFO:teuthology.orchestra.run.trial017.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release [7662 B] 2026-02-22T19:47:00.751 INFO:teuthology.orchestra.run.trial007.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T19:47:00.801 INFO:teuthology.orchestra.run.trial017.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-22T19:47:00.832 INFO:teuthology.orchestra.run.trial007.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-22T19:47:00.870 INFO:teuthology.orchestra.run.trial017.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2026-02-22T19:47:00.920 INFO:teuthology.orchestra.run.trial007.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-22T19:47:01.008 INFO:teuthology.orchestra.run.trial007.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-22T19:47:01.038 INFO:teuthology.orchestra.run.trial017.stdout:Fetched 25.8 kB in 1s (45.8 kB/s) 2026-02-22T19:47:01.560 INFO:teuthology.orchestra.run.trial017.stdout:Reading package lists... 2026-02-22T19:47:01.573 DEBUG:teuthology.orchestra.run.trial017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.2.3-689-gf90f4eae-1jammy cephadm=19.2.3-689-gf90f4eae-1jammy ceph-mds=19.2.3-689-gf90f4eae-1jammy ceph-mgr=19.2.3-689-gf90f4eae-1jammy ceph-common=19.2.3-689-gf90f4eae-1jammy ceph-fuse=19.2.3-689-gf90f4eae-1jammy ceph-test=19.2.3-689-gf90f4eae-1jammy ceph-volume=19.2.3-689-gf90f4eae-1jammy radosgw=19.2.3-689-gf90f4eae-1jammy python3-rados=19.2.3-689-gf90f4eae-1jammy python3-rgw=19.2.3-689-gf90f4eae-1jammy python3-cephfs=19.2.3-689-gf90f4eae-1jammy python3-rbd=19.2.3-689-gf90f4eae-1jammy libcephfs2=19.2.3-689-gf90f4eae-1jammy libcephfs-dev=19.2.3-689-gf90f4eae-1jammy librados2=19.2.3-689-gf90f4eae-1jammy librbd1=19.2.3-689-gf90f4eae-1jammy rbd-fuse=19.2.3-689-gf90f4eae-1jammy rbd-mirror=19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:47:01.606 INFO:teuthology.orchestra.run.trial017.stdout:Reading package lists... 2026-02-22T19:47:01.711 INFO:teuthology.orchestra.run.trial017.stdout:Building dependency tree... 2026-02-22T19:47:01.712 INFO:teuthology.orchestra.run.trial017.stdout:Reading state information... 2026-02-22T19:47:01.801 INFO:teuthology.orchestra.run.trial017.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T19:47:01.802 INFO:teuthology.orchestra.run.trial017.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T19:47:01.802 INFO:teuthology.orchestra.run.trial017.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T19:47:01.802 INFO:teuthology.orchestra.run.trial017.stdout: sg3-utils-udev 2026-02-22T19:47:01.802 INFO:teuthology.orchestra.run.trial017.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T19:47:01.802 INFO:teuthology.orchestra.run.trial017.stdout:The following additional packages will be installed: 2026-02-22T19:47:01.802 INFO:teuthology.orchestra.run.trial017.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-22T19:47:01.802 INFO:teuthology.orchestra.run.trial017.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-22T19:47:01.802 INFO:teuthology.orchestra.run.trial017.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 2026-02-22T19:47:01.802 INFO:teuthology.orchestra.run.trial017.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: pkg-config python-asyncssh-doc python-pastedeploy-tpl python3-asyncssh 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cheroot 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2026-02-22T19:47:01.803 INFO:teuthology.orchestra.run.trial017.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout: python3-simplejson python3-singledispatch python3-sklearn 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout: python3-toml python3-waitress python3-wcwidth python3-webob 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout:Suggested packages: 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout: python3-influxdb readline-doc python3-beaker python-mako-doc 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout: python-natsort-doc httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout: python-pygments-doc ttf-bitstream-vera python-pyinotify-doc python3-dap 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2026-02-22T19:47:01.804 INFO:teuthology.orchestra.run.trial017.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2026-02-22T19:47:01.805 INFO:teuthology.orchestra.run.trial017.stdout:Recommended packages: 2026-02-22T19:47:01.805 INFO:teuthology.orchestra.run.trial017.stdout: btrfs-tools 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout:The following NEW packages will be installed: 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: python-pastedeploy-tpl python3-asyncssh python3-cachetools 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cheroot 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T19:47:01.836 INFO:teuthology.orchestra.run.trial017.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T19:47:01.837 INFO:teuthology.orchestra.run.trial017.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T19:47:01.837 INFO:teuthology.orchestra.run.trial017.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T19:47:01.837 INFO:teuthology.orchestra.run.trial017.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2026-02-22T19:47:01.837 INFO:teuthology.orchestra.run.trial017.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2026-02-22T19:47:01.837 INFO:teuthology.orchestra.run.trial017.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2026-02-22T19:47:01.837 INFO:teuthology.orchestra.run.trial017.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2026-02-22T19:47:01.837 INFO:teuthology.orchestra.run.trial017.stdout: python3-threadpoolctl python3-toml python3-waitress python3-wcwidth 2026-02-22T19:47:01.837 INFO:teuthology.orchestra.run.trial017.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2026-02-22T19:47:01.837 INFO:teuthology.orchestra.run.trial017.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse rbd-mirror socat 2026-02-22T19:47:01.837 INFO:teuthology.orchestra.run.trial017.stdout: unzip xmlstarlet zip 2026-02-22T19:47:01.942 INFO:teuthology.orchestra.run.trial017.stdout:0 upgraded, 109 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T19:47:01.942 INFO:teuthology.orchestra.run.trial017.stdout:Need to get 181 MB of archives. 2026-02-22T19:47:01.942 INFO:teuthology.orchestra.run.trial017.stdout:After this operation, 821 MB of additional disk space will be used. 2026-02-22T19:47:01.942 INFO:teuthology.orchestra.run.trial017.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-22T19:47:01.988 INFO:teuthology.orchestra.run.trial017.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-22T19:47:01.989 INFO:teuthology.orchestra.run.trial017.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-22T19:47:01.998 INFO:teuthology.orchestra.run.trial017.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-22T19:47:02.028 INFO:teuthology.orchestra.run.trial017.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-22T19:47:02.030 INFO:teuthology.orchestra.run.trial017.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-22T19:47:02.035 INFO:teuthology.orchestra.run.trial017.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-22T19:47:02.037 INFO:teuthology.orchestra.run.trial017.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-22T19:47:02.037 INFO:teuthology.orchestra.run.trial017.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-22T19:47:02.038 INFO:teuthology.orchestra.run.trial017.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-22T19:47:02.038 INFO:teuthology.orchestra.run.trial017.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-22T19:47:02.042 INFO:teuthology.orchestra.run.trial017.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2026-02-22T19:47:02.044 INFO:teuthology.orchestra.run.trial017.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2026-02-22T19:47:02.045 INFO:teuthology.orchestra.run.trial017.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2026-02-22T19:47:02.049 INFO:teuthology.orchestra.run.trial017.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5034 B] 2026-02-22T19:47:02.049 INFO:teuthology.orchestra.run.trial017.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2026-02-22T19:47:02.050 INFO:teuthology.orchestra.run.trial017.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2026-02-22T19:47:02.052 INFO:teuthology.orchestra.run.trial017.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2026-02-22T19:47:02.053 INFO:teuthology.orchestra.run.trial017.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-22T19:47:02.058 INFO:teuthology.orchestra.run.trial017.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-22T19:47:02.059 INFO:teuthology.orchestra.run.trial017.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-22T19:47:02.059 INFO:teuthology.orchestra.run.trial017.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-22T19:47:02.059 INFO:teuthology.orchestra.run.trial017.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-22T19:47:02.059 INFO:teuthology.orchestra.run.trial017.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-22T19:47:02.068 INFO:teuthology.orchestra.run.trial017.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-22T19:47:02.068 INFO:teuthology.orchestra.run.trial017.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-22T19:47:02.068 INFO:teuthology.orchestra.run.trial017.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-22T19:47:02.069 INFO:teuthology.orchestra.run.trial017.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-22T19:47:02.070 INFO:teuthology.orchestra.run.trial017.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-22T19:47:02.077 INFO:teuthology.orchestra.run.trial017.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2026-02-22T19:47:02.078 INFO:teuthology.orchestra.run.trial017.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2026-02-22T19:47:02.078 INFO:teuthology.orchestra.run.trial017.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2026-02-22T19:47:02.079 INFO:teuthology.orchestra.run.trial017.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7320 B] 2026-02-22T19:47:02.079 INFO:teuthology.orchestra.run.trial017.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-22T19:47:02.087 INFO:teuthology.orchestra.run.trial017.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-waitress all 1.4.4-1.1ubuntu1.1 [47.0 kB] 2026-02-22T19:47:02.087 INFO:teuthology.orchestra.run.trial017.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2026-02-22T19:47:02.087 INFO:teuthology.orchestra.run.trial017.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2026-02-22T19:47:02.091 INFO:teuthology.orchestra.run.trial017.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4892 B] 2026-02-22T19:47:02.091 INFO:teuthology.orchestra.run.trial017.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2026-02-22T19:47:02.096 INFO:teuthology.orchestra.run.trial017.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2026-02-22T19:47:02.097 INFO:teuthology.orchestra.run.trial017.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2026-02-22T19:47:02.097 INFO:teuthology.orchestra.run.trial017.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.3 [181 kB] 2026-02-22T19:47:02.099 INFO:teuthology.orchestra.run.trial017.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-22T19:47:02.100 INFO:teuthology.orchestra.run.trial017.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-22T19:47:02.106 INFO:teuthology.orchestra.run.trial017.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-22T19:47:02.106 INFO:teuthology.orchestra.run.trial017.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-22T19:47:02.107 INFO:teuthology.orchestra.run.trial017.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-22T19:47:02.121 INFO:teuthology.orchestra.run.trial017.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-22T19:47:02.122 INFO:teuthology.orchestra.run.trial017.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-22T19:47:02.123 INFO:teuthology.orchestra.run.trial017.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-22T19:47:02.133 INFO:teuthology.orchestra.run.trial017.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-22T19:47:02.133 INFO:teuthology.orchestra.run.trial017.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-22T19:47:02.133 INFO:teuthology.orchestra.run.trial017.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-22T19:47:02.133 INFO:teuthology.orchestra.run.trial017.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-22T19:47:02.133 INFO:teuthology.orchestra.run.trial017.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-22T19:47:02.133 INFO:teuthology.orchestra.run.trial017.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-22T19:47:02.135 INFO:teuthology.orchestra.run.trial017.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-22T19:47:02.136 INFO:teuthology.orchestra.run.trial017.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-22T19:47:02.136 INFO:teuthology.orchestra.run.trial017.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-22T19:47:02.144 INFO:teuthology.orchestra.run.trial017.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-22T19:47:02.146 INFO:teuthology.orchestra.run.trial017.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-22T19:47:02.147 INFO:teuthology.orchestra.run.trial017.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2026-02-22T19:47:02.147 INFO:teuthology.orchestra.run.trial017.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2026-02-22T19:47:02.147 INFO:teuthology.orchestra.run.trial017.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-22T19:47:02.153 INFO:teuthology.orchestra.run.trial017.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2026-02-22T19:47:02.154 INFO:teuthology.orchestra.run.trial017.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-22T19:47:02.156 INFO:teuthology.orchestra.run.trial017.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-22T19:47:02.156 INFO:teuthology.orchestra.run.trial017.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2026-02-22T19:47:02.157 INFO:teuthology.orchestra.run.trial017.stdout:Get:69 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-22T19:47:02.163 INFO:teuthology.orchestra.run.trial017.stdout:Get:70 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-22T19:47:02.164 INFO:teuthology.orchestra.run.trial017.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-22T19:47:02.164 INFO:teuthology.orchestra.run.trial017.stdout:Get:72 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-22T19:47:02.168 INFO:teuthology.orchestra.run.trial017.stdout:Get:73 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2026-02-22T19:47:02.169 INFO:teuthology.orchestra.run.trial017.stdout:Get:74 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-22T19:47:02.173 INFO:teuthology.orchestra.run.trial017.stdout:Get:75 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-22T19:47:02.174 INFO:teuthology.orchestra.run.trial017.stdout:Get:76 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-22T19:47:02.175 INFO:teuthology.orchestra.run.trial017.stdout:Get:77 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-22T19:47:02.272 INFO:teuthology.orchestra.run.trial017.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.2.3-689-gf90f4eae-1jammy [3597 kB] 2026-02-22T19:47:02.713 INFO:teuthology.orchestra.run.trial007.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy InRelease 2026-02-22T19:47:02.777 INFO:teuthology.orchestra.run.trial007.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release [7662 B] 2026-02-22T19:47:02.843 INFO:teuthology.orchestra.run.trial007.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-22T19:47:02.910 INFO:teuthology.orchestra.run.trial007.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2026-02-22T19:47:03.067 INFO:teuthology.orchestra.run.trial007.stdout:Fetched 25.8 kB in 3s (9704 B/s) 2026-02-22T19:47:03.591 INFO:teuthology.orchestra.run.trial007.stdout:Reading package lists... 2026-02-22T19:47:03.605 DEBUG:teuthology.orchestra.run.trial007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.2.3-689-gf90f4eae-1jammy cephadm=19.2.3-689-gf90f4eae-1jammy ceph-mds=19.2.3-689-gf90f4eae-1jammy ceph-mgr=19.2.3-689-gf90f4eae-1jammy ceph-common=19.2.3-689-gf90f4eae-1jammy ceph-fuse=19.2.3-689-gf90f4eae-1jammy ceph-test=19.2.3-689-gf90f4eae-1jammy ceph-volume=19.2.3-689-gf90f4eae-1jammy radosgw=19.2.3-689-gf90f4eae-1jammy python3-rados=19.2.3-689-gf90f4eae-1jammy python3-rgw=19.2.3-689-gf90f4eae-1jammy python3-cephfs=19.2.3-689-gf90f4eae-1jammy python3-rbd=19.2.3-689-gf90f4eae-1jammy libcephfs2=19.2.3-689-gf90f4eae-1jammy libcephfs-dev=19.2.3-689-gf90f4eae-1jammy librados2=19.2.3-689-gf90f4eae-1jammy librbd1=19.2.3-689-gf90f4eae-1jammy rbd-fuse=19.2.3-689-gf90f4eae-1jammy rbd-mirror=19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:47:03.638 INFO:teuthology.orchestra.run.trial007.stdout:Reading package lists... 2026-02-22T19:47:03.746 INFO:teuthology.orchestra.run.trial007.stdout:Building dependency tree... 2026-02-22T19:47:03.746 INFO:teuthology.orchestra.run.trial007.stdout:Reading state information... 2026-02-22T19:47:03.837 INFO:teuthology.orchestra.run.trial007.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: sg3-utils-udev 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout:The following additional packages will be installed: 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: pkg-config python-asyncssh-doc python-pastedeploy-tpl python3-asyncssh 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cheroot 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-simplejson python3-singledispatch python3-sklearn 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2026-02-22T19:47:03.838 INFO:teuthology.orchestra.run.trial007.stdout: python3-toml python3-waitress python3-wcwidth python3-webob 2026-02-22T19:47:03.839 INFO:teuthology.orchestra.run.trial007.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2026-02-22T19:47:03.839 INFO:teuthology.orchestra.run.trial007.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2026-02-22T19:47:03.839 INFO:teuthology.orchestra.run.trial007.stdout:Suggested packages: 2026-02-22T19:47:03.839 INFO:teuthology.orchestra.run.trial007.stdout: python3-influxdb readline-doc python3-beaker python-mako-doc 2026-02-22T19:47:03.839 INFO:teuthology.orchestra.run.trial007.stdout: python-natsort-doc httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2026-02-22T19:47:03.839 INFO:teuthology.orchestra.run.trial007.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2026-02-22T19:47:03.839 INFO:teuthology.orchestra.run.trial007.stdout: python-pygments-doc ttf-bitstream-vera python-pyinotify-doc python3-dap 2026-02-22T19:47:03.839 INFO:teuthology.orchestra.run.trial007.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2026-02-22T19:47:03.839 INFO:teuthology.orchestra.run.trial007.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2026-02-22T19:47:03.839 INFO:teuthology.orchestra.run.trial007.stdout:Recommended packages: 2026-02-22T19:47:03.839 INFO:teuthology.orchestra.run.trial007.stdout: btrfs-tools 2026-02-22T19:47:03.873 INFO:teuthology.orchestra.run.trial007.stdout:The following NEW packages will be installed: 2026-02-22T19:47:03.873 INFO:teuthology.orchestra.run.trial007.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-22T19:47:03.873 INFO:teuthology.orchestra.run.trial007.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-22T19:47:03.873 INFO:teuthology.orchestra.run.trial007.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-22T19:47:03.873 INFO:teuthology.orchestra.run.trial007.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 2026-02-22T19:47:03.873 INFO:teuthology.orchestra.run.trial007.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-22T19:47:03.873 INFO:teuthology.orchestra.run.trial007.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python-pastedeploy-tpl python3-asyncssh python3-cachetools 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cheroot 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-threadpoolctl python3-toml python3-waitress python3-wcwidth 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse rbd-mirror socat 2026-02-22T19:47:03.874 INFO:teuthology.orchestra.run.trial007.stdout: unzip xmlstarlet zip 2026-02-22T19:47:03.978 INFO:teuthology.orchestra.run.trial007.stdout:0 upgraded, 109 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T19:47:03.978 INFO:teuthology.orchestra.run.trial007.stdout:Need to get 181 MB of archives. 2026-02-22T19:47:03.978 INFO:teuthology.orchestra.run.trial007.stdout:After this operation, 821 MB of additional disk space will be used. 2026-02-22T19:47:03.978 INFO:teuthology.orchestra.run.trial007.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-22T19:47:04.023 INFO:teuthology.orchestra.run.trial007.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-22T19:47:04.024 INFO:teuthology.orchestra.run.trial007.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-22T19:47:04.034 INFO:teuthology.orchestra.run.trial007.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-22T19:47:04.063 INFO:teuthology.orchestra.run.trial007.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-22T19:47:04.065 INFO:teuthology.orchestra.run.trial007.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-22T19:47:04.070 INFO:teuthology.orchestra.run.trial007.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-22T19:47:04.072 INFO:teuthology.orchestra.run.trial007.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-22T19:47:04.072 INFO:teuthology.orchestra.run.trial007.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-22T19:47:04.073 INFO:teuthology.orchestra.run.trial007.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-22T19:47:04.073 INFO:teuthology.orchestra.run.trial007.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-22T19:47:04.077 INFO:teuthology.orchestra.run.trial007.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2026-02-22T19:47:04.079 INFO:teuthology.orchestra.run.trial007.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2026-02-22T19:47:04.080 INFO:teuthology.orchestra.run.trial007.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2026-02-22T19:47:04.083 INFO:teuthology.orchestra.run.trial007.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5034 B] 2026-02-22T19:47:04.083 INFO:teuthology.orchestra.run.trial007.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2026-02-22T19:47:04.085 INFO:teuthology.orchestra.run.trial007.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2026-02-22T19:47:04.086 INFO:teuthology.orchestra.run.trial007.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2026-02-22T19:47:04.093 INFO:teuthology.orchestra.run.trial007.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-22T19:47:04.093 INFO:teuthology.orchestra.run.trial007.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-22T19:47:04.093 INFO:teuthology.orchestra.run.trial007.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-22T19:47:04.094 INFO:teuthology.orchestra.run.trial007.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-22T19:47:04.094 INFO:teuthology.orchestra.run.trial007.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-22T19:47:04.094 INFO:teuthology.orchestra.run.trial007.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-22T19:47:04.102 INFO:teuthology.orchestra.run.trial007.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-22T19:47:04.102 INFO:teuthology.orchestra.run.trial007.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-22T19:47:04.102 INFO:teuthology.orchestra.run.trial007.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-22T19:47:04.103 INFO:teuthology.orchestra.run.trial007.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-22T19:47:04.111 INFO:teuthology.orchestra.run.trial007.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-22T19:47:04.112 INFO:teuthology.orchestra.run.trial007.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2026-02-22T19:47:04.112 INFO:teuthology.orchestra.run.trial007.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2026-02-22T19:47:04.113 INFO:teuthology.orchestra.run.trial007.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2026-02-22T19:47:04.113 INFO:teuthology.orchestra.run.trial007.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7320 B] 2026-02-22T19:47:04.113 INFO:teuthology.orchestra.run.trial007.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-22T19:47:04.121 INFO:teuthology.orchestra.run.trial007.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-waitress all 1.4.4-1.1ubuntu1.1 [47.0 kB] 2026-02-22T19:47:04.121 INFO:teuthology.orchestra.run.trial007.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2026-02-22T19:47:04.122 INFO:teuthology.orchestra.run.trial007.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2026-02-22T19:47:04.125 INFO:teuthology.orchestra.run.trial007.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4892 B] 2026-02-22T19:47:04.130 INFO:teuthology.orchestra.run.trial007.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2026-02-22T19:47:04.130 INFO:teuthology.orchestra.run.trial007.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2026-02-22T19:47:04.131 INFO:teuthology.orchestra.run.trial007.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2026-02-22T19:47:04.132 INFO:teuthology.orchestra.run.trial007.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.3 [181 kB] 2026-02-22T19:47:04.133 INFO:teuthology.orchestra.run.trial007.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-22T19:47:04.134 INFO:teuthology.orchestra.run.trial007.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-22T19:47:04.140 INFO:teuthology.orchestra.run.trial007.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-22T19:47:04.140 INFO:teuthology.orchestra.run.trial007.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-22T19:47:04.141 INFO:teuthology.orchestra.run.trial007.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-22T19:47:04.155 INFO:teuthology.orchestra.run.trial007.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-22T19:47:04.156 INFO:teuthology.orchestra.run.trial007.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-22T19:47:04.157 INFO:teuthology.orchestra.run.trial007.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-22T19:47:04.167 INFO:teuthology.orchestra.run.trial007.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-22T19:47:04.167 INFO:teuthology.orchestra.run.trial007.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-22T19:47:04.167 INFO:teuthology.orchestra.run.trial007.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-22T19:47:04.167 INFO:teuthology.orchestra.run.trial007.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-22T19:47:04.167 INFO:teuthology.orchestra.run.trial007.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-22T19:47:04.168 INFO:teuthology.orchestra.run.trial007.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-22T19:47:04.169 INFO:teuthology.orchestra.run.trial007.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-22T19:47:04.170 INFO:teuthology.orchestra.run.trial007.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-22T19:47:04.170 INFO:teuthology.orchestra.run.trial007.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-22T19:47:04.178 INFO:teuthology.orchestra.run.trial007.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-22T19:47:04.180 INFO:teuthology.orchestra.run.trial007.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-22T19:47:04.181 INFO:teuthology.orchestra.run.trial007.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2026-02-22T19:47:04.182 INFO:teuthology.orchestra.run.trial007.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2026-02-22T19:47:04.182 INFO:teuthology.orchestra.run.trial007.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-22T19:47:04.184 INFO:teuthology.orchestra.run.trial007.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2026-02-22T19:47:04.188 INFO:teuthology.orchestra.run.trial007.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-22T19:47:04.189 INFO:teuthology.orchestra.run.trial007.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-22T19:47:04.189 INFO:teuthology.orchestra.run.trial007.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2026-02-22T19:47:04.190 INFO:teuthology.orchestra.run.trial007.stdout:Get:69 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-22T19:47:04.197 INFO:teuthology.orchestra.run.trial007.stdout:Get:70 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-22T19:47:04.198 INFO:teuthology.orchestra.run.trial007.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-22T19:47:04.198 INFO:teuthology.orchestra.run.trial007.stdout:Get:72 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-22T19:47:04.202 INFO:teuthology.orchestra.run.trial007.stdout:Get:73 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2026-02-22T19:47:04.202 INFO:teuthology.orchestra.run.trial007.stdout:Get:74 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-22T19:47:04.202 INFO:teuthology.orchestra.run.trial007.stdout:Get:75 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-22T19:47:04.207 INFO:teuthology.orchestra.run.trial007.stdout:Get:76 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-22T19:47:04.207 INFO:teuthology.orchestra.run.trial007.stdout:Get:77 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-22T19:47:04.241 INFO:teuthology.orchestra.run.trial007.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.2.3-689-gf90f4eae-1jammy [3597 kB] 2026-02-22T19:47:09.121 INFO:teuthology.orchestra.run.trial007.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.2.3-689-gf90f4eae-1jammy [3256 kB] 2026-02-22T19:47:09.750 INFO:teuthology.orchestra.run.trial017.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.2.3-689-gf90f4eae-1jammy [3256 kB] 2026-02-22T19:47:13.994 INFO:teuthology.orchestra.run.trial007.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.2.3-689-gf90f4eae-1jammy [979 kB] 2026-02-22T19:47:15.003 INFO:teuthology.orchestra.run.trial017.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.2.3-689-gf90f4eae-1jammy [979 kB] 2026-02-22T19:47:15.309 INFO:teuthology.orchestra.run.trial007.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.2.3-689-gf90f4eae-1jammy [357 kB] 2026-02-22T19:47:15.753 INFO:teuthology.orchestra.run.trial007.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.2.3-689-gf90f4eae-1jammy [32.9 kB] 2026-02-22T19:47:15.807 INFO:teuthology.orchestra.run.trial007.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.2.3-689-gf90f4eae-1jammy [184 kB] 2026-02-22T19:47:16.077 INFO:teuthology.orchestra.run.trial007.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.2.3-689-gf90f4eae-1jammy [70.0 kB] 2026-02-22T19:47:16.166 INFO:teuthology.orchestra.run.trial007.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.2.3-689-gf90f4eae-1jammy [335 kB] 2026-02-22T19:47:16.791 INFO:teuthology.orchestra.run.trial007.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.2.3-689-gf90f4eae-1jammy [6935 kB] 2026-02-22T19:47:17.169 INFO:teuthology.orchestra.run.trial017.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.2.3-689-gf90f4eae-1jammy [357 kB] 2026-02-22T19:47:17.813 INFO:teuthology.orchestra.run.trial017.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.2.3-689-gf90f4eae-1jammy [32.9 kB] 2026-02-22T19:47:17.844 INFO:teuthology.orchestra.run.trial017.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.2.3-689-gf90f4eae-1jammy [184 kB] 2026-02-22T19:47:18.147 INFO:teuthology.orchestra.run.trial017.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.2.3-689-gf90f4eae-1jammy [70.0 kB] 2026-02-22T19:47:18.251 INFO:teuthology.orchestra.run.trial017.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.2.3-689-gf90f4eae-1jammy [335 kB] 2026-02-22T19:47:18.870 INFO:teuthology.orchestra.run.trial017.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.2.3-689-gf90f4eae-1jammy [6935 kB] 2026-02-22T19:47:26.887 INFO:teuthology.orchestra.run.trial017.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.2.3-689-gf90f4eae-1jammy [112 kB] 2026-02-22T19:47:27.024 INFO:teuthology.orchestra.run.trial017.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.2.3-689-gf90f4eae-1jammy [470 kB] 2026-02-22T19:47:27.546 INFO:teuthology.orchestra.run.trial017.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.2.3-689-gf90f4eae-1jammy [26.5 MB] 2026-02-22T19:47:30.173 INFO:teuthology.orchestra.run.trial007.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.2.3-689-gf90f4eae-1jammy [112 kB] 2026-02-22T19:47:30.519 INFO:teuthology.orchestra.run.trial007.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.2.3-689-gf90f4eae-1jammy [470 kB] 2026-02-22T19:47:32.420 INFO:teuthology.orchestra.run.trial007.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.2.3-689-gf90f4eae-1jammy [26.5 MB] 2026-02-22T19:48:05.092 INFO:teuthology.orchestra.run.trial017.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.2.3-689-gf90f4eae-1jammy [5178 kB] 2026-02-22T19:48:10.964 INFO:teuthology.orchestra.run.trial007.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.2.3-689-gf90f4eae-1jammy [5178 kB] 2026-02-22T19:48:18.738 INFO:teuthology.orchestra.run.trial017.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.2.3-689-gf90f4eae-1jammy [248 kB] 2026-02-22T19:48:19.218 INFO:teuthology.orchestra.run.trial017.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.2.3-689-gf90f4eae-1jammy [125 kB] 2026-02-22T19:48:19.459 INFO:teuthology.orchestra.run.trial017.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.2.3-689-gf90f4eae-1jammy [1081 kB] 2026-02-22T19:48:21.985 INFO:teuthology.orchestra.run.trial017.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.2.3-689-gf90f4eae-1jammy [6239 kB] 2026-02-22T19:48:25.831 INFO:teuthology.orchestra.run.trial007.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.2.3-689-gf90f4eae-1jammy [248 kB] 2026-02-22T19:48:26.805 INFO:teuthology.orchestra.run.trial007.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.2.3-689-gf90f4eae-1jammy [125 kB] 2026-02-22T19:48:27.293 INFO:teuthology.orchestra.run.trial007.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.2.3-689-gf90f4eae-1jammy [1081 kB] 2026-02-22T19:48:30.945 INFO:teuthology.orchestra.run.trial007.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.2.3-689-gf90f4eae-1jammy [6239 kB] 2026-02-22T19:48:37.930 INFO:teuthology.orchestra.run.trial017.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.2.3-689-gf90f4eae-1jammy [23.0 MB] 2026-02-22T19:48:47.447 INFO:teuthology.orchestra.run.trial007.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.2.3-689-gf90f4eae-1jammy [23.0 MB] 2026-02-22T19:49:09.467 INFO:teuthology.orchestra.run.trial017.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.2.3-689-gf90f4eae-1jammy [14.2 kB] 2026-02-22T19:49:09.501 INFO:teuthology.orchestra.run.trial017.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.2.3-689-gf90f4eae-1jammy [1173 kB] 2026-02-22T19:49:11.397 INFO:teuthology.orchestra.run.trial017.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.2.3-689-gf90f4eae-1jammy [2503 kB] 2026-02-22T19:49:14.339 INFO:teuthology.orchestra.run.trial017.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.2.3-689-gf90f4eae-1jammy [798 kB] 2026-02-22T19:49:15.241 INFO:teuthology.orchestra.run.trial017.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.2.3-689-gf90f4eae-1jammy [157 kB] 2026-02-22T19:49:15.380 INFO:teuthology.orchestra.run.trial017.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.2.3-689-gf90f4eae-1jammy [2396 kB] 2026-02-22T19:49:18.483 INFO:teuthology.orchestra.run.trial017.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.2.3-689-gf90f4eae-1jammy [8625 kB] 2026-02-22T19:49:26.450 INFO:teuthology.orchestra.run.trial007.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.2.3-689-gf90f4eae-1jammy [14.2 kB] 2026-02-22T19:49:26.450 INFO:teuthology.orchestra.run.trial007.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.2.3-689-gf90f4eae-1jammy [1173 kB] 2026-02-22T19:49:27.378 INFO:teuthology.orchestra.run.trial017.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.2.3-689-gf90f4eae-1jammy [14.3 kB] 2026-02-22T19:49:27.380 INFO:teuthology.orchestra.run.trial017.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.2.3-689-gf90f4eae-1jammy [52.1 MB] 2026-02-22T19:49:27.745 INFO:teuthology.orchestra.run.trial007.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.2.3-689-gf90f4eae-1jammy [2503 kB] 2026-02-22T19:49:30.606 INFO:teuthology.orchestra.run.trial007.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.2.3-689-gf90f4eae-1jammy [798 kB] 2026-02-22T19:49:31.518 INFO:teuthology.orchestra.run.trial007.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.2.3-689-gf90f4eae-1jammy [157 kB] 2026-02-22T19:49:31.685 INFO:teuthology.orchestra.run.trial007.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.2.3-689-gf90f4eae-1jammy [2396 kB] 2026-02-22T19:49:36.165 INFO:teuthology.orchestra.run.trial007.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.2.3-689-gf90f4eae-1jammy [8625 kB] 2026-02-22T19:49:48.732 INFO:teuthology.orchestra.run.trial007.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.2.3-689-gf90f4eae-1jammy [14.3 kB] 2026-02-22T19:49:48.750 INFO:teuthology.orchestra.run.trial007.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.2.3-689-gf90f4eae-1jammy [52.1 MB] 2026-02-22T19:50:17.305 INFO:teuthology.orchestra.run.trial017.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.2.3-689-gf90f4eae-1jammy [135 kB] 2026-02-22T19:50:17.373 INFO:teuthology.orchestra.run.trial017.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.2.3-689-gf90f4eae-1jammy [41.0 kB] 2026-02-22T19:50:17.408 INFO:teuthology.orchestra.run.trial017.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.2.3-689-gf90f4eae-1jammy [13.7 MB] 2026-02-22T19:50:28.017 INFO:teuthology.orchestra.run.trial017.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.2.3-689-gf90f4eae-1jammy [92.3 kB] 2026-02-22T19:50:28.087 INFO:teuthology.orchestra.run.trial017.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.2.3-689-gf90f4eae-1jammy [3151 kB] 2026-02-22T19:50:30.318 INFO:teuthology.orchestra.run.trial017.stdout:Fetched 181 MB in 3min 28s (868 kB/s) 2026-02-22T19:50:30.335 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-22T19:50:30.496 INFO:teuthology.orchestra.run.trial017.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-22T19:50:30.498 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-22T19:50:30.500 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T19:50:30.520 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-22T19:50:30.528 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-22T19:50:30.529 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T19:50:30.544 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-22T19:50:30.556 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-22T19:50:30.556 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T19:50:30.575 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-22T19:50:30.583 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:30.588 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:30.620 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-22T19:50:30.628 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:30.629 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:30.646 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-22T19:50:30.654 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:30.654 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:30.675 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-22T19:50:30.683 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-22T19:50:30.683 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T19:50:30.700 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package librados2. 2026-02-22T19:50:30.708 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../007-librados2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:30.708 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:30.748 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libnbd0. 2026-02-22T19:50:30.757 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2026-02-22T19:50:30.757 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-22T19:50:30.771 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package librbd1. 2026-02-22T19:50:30.779 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../009-librbd1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:30.780 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:30.819 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libcephfs2. 2026-02-22T19:50:30.828 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../010-libcephfs2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:30.828 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:30.849 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-rados. 2026-02-22T19:50:30.857 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../011-python3-rados_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:30.857 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:30.874 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-22T19:50:30.883 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:30.883 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:30.897 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-cephfs. 2026-02-22T19:50:30.905 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../013-python3-cephfs_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:30.906 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:30.920 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-22T19:50:30.929 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../014-python3-ceph-common_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:30.929 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:30.947 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-22T19:50:30.955 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-22T19:50:30.956 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T19:50:30.971 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-prettytable. 2026-02-22T19:50:30.980 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2026-02-22T19:50:30.980 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-22T19:50:30.994 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-rbd. 2026-02-22T19:50:31.002 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../017-python3-rbd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:31.002 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:31.019 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-22T19:50:31.028 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-22T19:50:31.028 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T19:50:31.046 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libreadline-dev:amd64. 2026-02-22T19:50:31.061 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2026-02-22T19:50:31.061 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T19:50:31.072 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2026-02-22T19:50:31.080 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2026-02-22T19:50:31.080 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T19:50:31.098 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package lua5.1. 2026-02-22T19:50:31.105 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2026-02-22T19:50:31.106 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2026-02-22T19:50:31.125 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package lua-any. 2026-02-22T19:50:31.134 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2026-02-22T19:50:31.134 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking lua-any (27ubuntu1) ... 2026-02-22T19:50:31.148 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package zip. 2026-02-22T19:50:31.156 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2026-02-22T19:50:31.156 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking zip (3.0-12build2) ... 2026-02-22T19:50:31.173 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package unzip. 2026-02-22T19:50:31.181 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2026-02-22T19:50:31.181 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2026-02-22T19:50:31.199 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package luarocks. 2026-02-22T19:50:31.207 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2026-02-22T19:50:31.208 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2026-02-22T19:50:31.244 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package librgw2. 2026-02-22T19:50:31.252 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../026-librgw2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:31.253 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:31.323 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-rgw. 2026-02-22T19:50:31.331 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../027-python3-rgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:31.332 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:31.347 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-22T19:50:31.355 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../028-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-22T19:50:31.355 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T19:50:31.370 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libradosstriper1. 2026-02-22T19:50:31.378 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../029-libradosstriper1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:31.378 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:31.397 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-common. 2026-02-22T19:50:31.404 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../030-ceph-common_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:31.405 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:31.606 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-base. 2026-02-22T19:50:31.615 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../031-ceph-base_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:31.620 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:31.684 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-22T19:50:31.692 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-22T19:50:31.692 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-22T19:50:31.707 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-cheroot. 2026-02-22T19:50:31.715 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-22T19:50:31.716 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T19:50:31.732 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-22T19:50:31.740 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-22T19:50:31.741 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-22T19:50:31.755 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-22T19:50:31.764 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-22T19:50:31.764 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-22T19:50:31.777 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-22T19:50:31.786 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-22T19:50:31.786 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-22T19:50:31.800 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-tempora. 2026-02-22T19:50:31.808 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2026-02-22T19:50:31.809 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-22T19:50:31.824 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-portend. 2026-02-22T19:50:31.832 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2026-02-22T19:50:31.832 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-22T19:50:31.847 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-22T19:50:31.855 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-22T19:50:31.855 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-22T19:50:31.871 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-22T19:50:31.879 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-22T19:50:31.880 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-22T19:50:31.902 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-natsort. 2026-02-22T19:50:31.910 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2026-02-22T19:50:31.911 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-22T19:50:31.927 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-logutils. 2026-02-22T19:50:31.935 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2026-02-22T19:50:31.935 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-logutils (0.3.3-8) ... 2026-02-22T19:50:31.950 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-mako. 2026-02-22T19:50:31.958 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2026-02-22T19:50:31.958 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T19:50:31.976 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-simplegeneric. 2026-02-22T19:50:31.984 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2026-02-22T19:50:31.984 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2026-02-22T19:50:31.999 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-singledispatch. 2026-02-22T19:50:32.007 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2026-02-22T19:50:32.008 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2026-02-22T19:50:32.021 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-webob. 2026-02-22T19:50:32.029 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-22T19:50:32.030 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T19:50:32.046 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-waitress. 2026-02-22T19:50:32.054 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1.1_all.deb ... 2026-02-22T19:50:32.056 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T19:50:32.072 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-tempita. 2026-02-22T19:50:32.080 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2026-02-22T19:50:32.081 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T19:50:32.095 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-paste. 2026-02-22T19:50:32.103 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2026-02-22T19:50:32.104 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T19:50:32.131 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2026-02-22T19:50:32.139 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2026-02-22T19:50:32.140 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T19:50:32.154 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-pastedeploy. 2026-02-22T19:50:32.162 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2026-02-22T19:50:32.162 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2026-02-22T19:50:32.176 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-webtest. 2026-02-22T19:50:32.184 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2026-02-22T19:50:32.185 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-webtest (2.0.35-1) ... 2026-02-22T19:50:32.200 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-pecan. 2026-02-22T19:50:32.208 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2026-02-22T19:50:32.208 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T19:50:32.232 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-werkzeug. 2026-02-22T19:50:32.240 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.3_all.deb ... 2026-02-22T19:50:32.241 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T19:50:32.260 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-22T19:50:32.268 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:32.269 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:32.297 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-22T19:50:32.305 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:32.305 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:32.319 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-mgr. 2026-02-22T19:50:32.328 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../057-ceph-mgr_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:32.328 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:32.350 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-mon. 2026-02-22T19:50:32.358 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../058-ceph-mon_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:32.359 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:32.414 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-22T19:50:32.423 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../059-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-22T19:50:32.423 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T19:50:32.439 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-osd. 2026-02-22T19:50:32.447 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../060-ceph-osd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:32.448 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:32.601 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph. 2026-02-22T19:50:32.610 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../061-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:32.610 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:32.624 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-fuse. 2026-02-22T19:50:32.632 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../062-ceph-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:32.633 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:32.655 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-mds. 2026-02-22T19:50:32.664 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../063-ceph-mds_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:32.664 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:32.696 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package cephadm. 2026-02-22T19:50:32.704 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../064-cephadm_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:32.705 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:32.720 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-22T19:50:32.728 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T19:50:32.729 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:32.750 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-22T19:50:32.758 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:32.758 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:32.779 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-22T19:50:32.787 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../067-python3-repoze.lru_0.7-2_all.deb ... 2026-02-22T19:50:32.787 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-22T19:50:32.802 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-routes. 2026-02-22T19:50:32.811 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../068-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-22T19:50:32.811 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T19:50:32.830 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-22T19:50:32.839 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../069-ceph-mgr-dashboard_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:32.839 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:33.041 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-22T19:50:33.050 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../070-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-22T19:50:33.050 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T19:50:33.088 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-joblib. 2026-02-22T19:50:33.096 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../071-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-22T19:50:33.097 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T19:50:33.124 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-22T19:50:33.132 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../072-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-22T19:50:33.133 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-22T19:50:33.147 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-sklearn. 2026-02-22T19:50:33.156 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../073-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-22T19:50:33.156 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T19:50:33.237 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-22T19:50:33.246 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../074-ceph-mgr-diskprediction-local_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:33.247 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:33.413 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-cachetools. 2026-02-22T19:50:33.422 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../075-python3-cachetools_5.0.0-1_all.deb ... 2026-02-22T19:50:33.422 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-22T19:50:33.437 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-rsa. 2026-02-22T19:50:33.445 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../076-python3-rsa_4.8-1_all.deb ... 2026-02-22T19:50:33.446 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-22T19:50:33.463 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-google-auth. 2026-02-22T19:50:33.472 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-3_all.deb ... 2026-02-22T19:50:33.472 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-22T19:50:33.489 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-22T19:50:33.498 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-22T19:50:33.498 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T19:50:33.514 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-websocket. 2026-02-22T19:50:33.522 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../079-python3-websocket_1.2.3-1_all.deb ... 2026-02-22T19:50:33.522 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-22T19:50:33.540 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-22T19:50:33.548 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../080-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-22T19:50:33.549 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T19:50:33.641 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-22T19:50:33.650 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:33.651 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:33.665 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-22T19:50:33.674 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../082-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-22T19:50:33.681 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T19:50:33.696 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-22T19:50:33.705 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../083-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T19:50:33.706 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T19:50:33.720 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package jq. 2026-02-22T19:50:33.728 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../084-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T19:50:33.729 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-22T19:50:33.743 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package socat. 2026-02-22T19:50:33.751 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../085-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-22T19:50:33.752 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-22T19:50:33.772 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package xmlstarlet. 2026-02-22T19:50:33.780 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-22T19:50:33.780 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-22T19:50:33.816 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-test. 2026-02-22T19:50:33.824 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../087-ceph-test_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:33.824 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:34.301 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package ceph-volume. 2026-02-22T19:50:34.310 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../088-ceph-volume_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:34.310 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:34.333 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-22T19:50:34.342 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../089-libcephfs-dev_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:34.342 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:34.356 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package lua-socket:amd64. 2026-02-22T19:50:34.365 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../090-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2026-02-22T19:50:34.366 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T19:50:34.386 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package lua-sec:amd64. 2026-02-22T19:50:34.395 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../091-lua-sec_1.0.2-1_amd64.deb ... 2026-02-22T19:50:34.395 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2026-02-22T19:50:34.412 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package nvme-cli. 2026-02-22T19:50:34.421 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../092-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-22T19:50:34.421 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T19:50:34.452 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package pkg-config. 2026-02-22T19:50:34.461 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../093-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2026-02-22T19:50:34.461 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T19:50:34.476 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-22T19:50:34.485 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T19:50:34.486 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:34.515 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-22T19:50:34.523 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-22T19:50:34.523 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-22T19:50:34.539 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-pastescript. 2026-02-22T19:50:34.547 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2026-02-22T19:50:34.548 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2026-02-22T19:50:34.566 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-pluggy. 2026-02-22T19:50:34.574 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-22T19:50:34.574 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-22T19:50:34.590 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-psutil. 2026-02-22T19:50:34.599 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-22T19:50:34.599 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-22T19:50:34.617 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-py. 2026-02-22T19:50:34.626 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2026-02-22T19:50:34.626 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-22T19:50:34.647 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-pygments. 2026-02-22T19:50:34.655 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-22T19:50:34.656 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T19:50:34.696 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-pyinotify. 2026-02-22T19:50:34.705 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2026-02-22T19:50:34.705 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2026-02-22T19:50:34.721 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-toml. 2026-02-22T19:50:34.729 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2026-02-22T19:50:34.729 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-22T19:50:34.744 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-pytest. 2026-02-22T19:50:34.753 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-22T19:50:34.753 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T19:50:34.775 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package python3-simplejson. 2026-02-22T19:50:34.784 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-22T19:50:34.784 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-22T19:50:34.801 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-22T19:50:34.809 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-22T19:50:34.810 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-22T19:50:34.877 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package radosgw. 2026-02-22T19:50:34.886 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../106-radosgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:34.886 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:34.999 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package rbd-fuse. 2026-02-22T19:50:35.007 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../107-rbd-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:35.008 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:35.023 INFO:teuthology.orchestra.run.trial017.stdout:Selecting previously unselected package rbd-mirror. 2026-02-22T19:50:35.032 INFO:teuthology.orchestra.run.trial017.stdout:Preparing to unpack .../108-rbd-mirror_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:35.032 INFO:teuthology.orchestra.run.trial017.stdout:Unpacking rbd-mirror (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:35.092 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-22T19:50:35.159 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T19:50:35.160 INFO:teuthology.orchestra.run.trial017.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T19:50:35.231 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T19:50:35.518 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-22T19:50:35.900 INFO:teuthology.orchestra.run.trial017.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-22T19:50:35.911 INFO:teuthology.orchestra.run.trial017.stdout:Setting up cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:35.953 INFO:teuthology.orchestra.run.trial017.stdout:Adding system user cephadm....done 2026-02-22T19:50:35.964 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T19:50:36.034 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-22T19:50:36.106 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:36.107 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-22T19:50:36.180 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-22T19:50:36.254 INFO:teuthology.orchestra.run.trial017.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T19:50:36.255 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-22T19:50:36.352 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T19:50:36.477 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-22T19:50:36.550 INFO:teuthology.orchestra.run.trial017.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2026-02-22T19:50:36.563 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2026-02-22T19:50:36.640 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-22T19:50:36.713 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:36.782 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T19:50:36.783 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-22T19:50:36.784 INFO:teuthology.orchestra.run.trial017.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T19:50:36.785 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T19:50:36.785 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T19:50:36.786 INFO:teuthology.orchestra.run.trial017.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2026-02-22T19:50:36.803 INFO:teuthology.orchestra.run.trial017.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2026-02-22T19:50:36.805 INFO:teuthology.orchestra.run.trial017.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2026-02-22T19:50:36.806 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T19:50:36.807 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-22T19:50:36.932 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-22T19:50:37.009 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T19:50:37.085 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-22T19:50:37.173 INFO:teuthology.orchestra.run.trial017.stdout:Setting up zip (3.0-12build2) ... 2026-02-22T19:50:37.174 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T19:50:37.424 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T19:50:37.499 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T19:50:37.500 INFO:teuthology.orchestra.run.trial017.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-22T19:50:37.501 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T19:50:37.597 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:37.721 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T19:50:37.851 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T19:50:37.936 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T19:50:38.041 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-22T19:50:38.115 INFO:teuthology.orchestra.run.trial017.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-22T19:50:38.116 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:38.198 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T19:50:38.697 INFO:teuthology.orchestra.run.trial017.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T19:50:38.719 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:38.724 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-22T19:50:38.799 INFO:teuthology.orchestra.run.trial017.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T19:50:38.800 INFO:teuthology.orchestra.run.trial017.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-22T19:50:38.801 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-22T19:50:38.879 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-22T19:50:38.952 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:38.953 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-22T19:50:39.035 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2026-02-22T19:50:39.111 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-logutils (0.3.3-8) ... 2026-02-22T19:50:39.187 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-22T19:50:39.264 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2026-02-22T19:50:39.326 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-22T19:50:39.409 INFO:teuthology.orchestra.run.trial017.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T19:50:39.410 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-22T19:50:39.495 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T19:50:39.496 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T19:50:39.575 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T19:50:39.653 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T19:50:39.738 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-22T19:50:39.813 INFO:teuthology.orchestra.run.trial017.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T19:50:39.814 INFO:teuthology.orchestra.run.trial017.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2026-02-22T19:50:39.814 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T19:50:39.815 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T19:50:39.951 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2026-02-22T19:50:40.029 INFO:teuthology.orchestra.run.trial017.stdout:Setting up lua-any (27ubuntu1) ... 2026-02-22T19:50:40.030 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-22T19:50:40.107 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:40.108 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-22T19:50:40.192 INFO:teuthology.orchestra.run.trial017.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-22T19:50:40.193 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-webtest (2.0.35-1) ... 2026-02-22T19:50:40.275 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-22T19:50:40.408 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-pastescript (2.0.2-4) ... 2026-02-22T19:50:40.499 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T19:50:40.616 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T19:50:40.617 INFO:teuthology.orchestra.run.trial017.stdout:Setting up librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:40.618 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:40.619 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T19:50:41.133 INFO:teuthology.orchestra.run.trial017.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2026-02-22T19:50:41.146 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.147 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.148 INFO:teuthology.orchestra.run.trial017.stdout:Setting up librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.149 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.150 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.218 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T19:50:41.218 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T19:50:41.620 INFO:teuthology.orchestra.run.trial017.stdout:Setting up libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.621 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.622 INFO:teuthology.orchestra.run.trial017.stdout:Setting up librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.622 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.623 INFO:teuthology.orchestra.run.trial017.stdout:Setting up rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.624 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.625 INFO:teuthology.orchestra.run.trial017.stdout:Setting up python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.626 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:41.654 INFO:teuthology.orchestra.run.trial017.stdout:Adding group ceph....done 2026-02-22T19:50:41.690 INFO:teuthology.orchestra.run.trial017.stdout:Adding system user ceph....done 2026-02-22T19:50:41.700 INFO:teuthology.orchestra.run.trial017.stdout:Setting system user ceph properties....done 2026-02-22T19:50:41.706 INFO:teuthology.orchestra.run.trial017.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-22T19:50:41.793 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-22T19:50:42.069 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-22T19:50:42.468 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:42.469 INFO:teuthology.orchestra.run.trial017.stdout:Setting up radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:42.720 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T19:50:42.720 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T19:50:43.114 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:43.222 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-22T19:50:43.606 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:43.666 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T19:50:43.667 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T19:50:44.046 INFO:teuthology.orchestra.run.trial017.stdout:Setting up rbd-mirror (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:44.121 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T19:50:44.122 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T19:50:44.509 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:44.587 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T19:50:44.587 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T19:50:44.960 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:45.057 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T19:50:45.057 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T19:50:45.453 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:45.454 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:45.465 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:45.536 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T19:50:45.536 INFO:teuthology.orchestra.run.trial017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T19:50:45.925 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:45.936 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:45.937 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:45.947 INFO:teuthology.orchestra.run.trial017.stdout:Setting up ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:46.058 INFO:teuthology.orchestra.run.trial017.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2026-02-22T19:50:46.066 INFO:teuthology.orchestra.run.trial017.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-22T19:50:46.309 INFO:teuthology.orchestra.run.trial017.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-22T19:50:46.395 INFO:teuthology.orchestra.run.trial017.stdout:Processing triggers for install-info (6.8-4build1) ... 2026-02-22T19:50:46.890 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:50:46.890 INFO:teuthology.orchestra.run.trial017.stdout:Running kernel seems to be up-to-date. 2026-02-22T19:50:46.890 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:50:46.890 INFO:teuthology.orchestra.run.trial017.stdout:The processor microcode seems to be up-to-date. 2026-02-22T19:50:46.891 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:50:46.891 INFO:teuthology.orchestra.run.trial017.stdout:No services need to be restarted. 2026-02-22T19:50:46.891 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:50:46.891 INFO:teuthology.orchestra.run.trial017.stdout:No containers need to be restarted. 2026-02-22T19:50:46.891 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:50:46.891 INFO:teuthology.orchestra.run.trial017.stdout:No user sessions are running outdated binaries. 2026-02-22T19:50:46.891 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:50:46.891 INFO:teuthology.orchestra.run.trial017.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-22T19:50:47.682 INFO:teuthology.orchestra.run.trial017.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-22T19:50:47.689 DEBUG:teuthology.parallel:result is None 2026-02-22T19:50:47.908 INFO:teuthology.orchestra.run.trial007.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.2.3-689-gf90f4eae-1jammy [135 kB] 2026-02-22T19:50:47.970 INFO:teuthology.orchestra.run.trial007.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.2.3-689-gf90f4eae-1jammy [41.0 kB] 2026-02-22T19:50:47.972 INFO:teuthology.orchestra.run.trial007.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.2.3-689-gf90f4eae-1jammy [13.7 MB] 2026-02-22T19:50:54.003 INFO:teuthology.orchestra.run.trial007.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.2.3-689-gf90f4eae-1jammy [92.3 kB] 2026-02-22T19:50:54.036 INFO:teuthology.orchestra.run.trial007.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.2.3-689-gf90f4eae-1jammy [3151 kB] 2026-02-22T19:50:55.145 INFO:teuthology.orchestra.run.trial007.stdout:Fetched 181 MB in 3min 51s (782 kB/s) 2026-02-22T19:50:55.161 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-22T19:50:55.317 INFO:teuthology.orchestra.run.trial007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-22T19:50:55.319 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-22T19:50:55.321 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T19:50:55.342 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-22T19:50:55.355 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-22T19:50:55.356 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T19:50:55.375 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-22T19:50:55.386 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-22T19:50:55.387 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T19:50:55.407 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-22T19:50:55.415 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:55.420 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:55.450 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-22T19:50:55.458 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:55.458 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:55.474 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-22T19:50:55.483 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T19:50:55.483 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:50:55.501 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-22T19:50:55.509 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-22T19:50:55.510 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T19:50:55.526 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package librados2. 2026-02-22T19:50:55.535 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../007-librados2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.535 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.576 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libnbd0. 2026-02-22T19:50:55.584 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2026-02-22T19:50:55.584 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-22T19:50:55.599 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package librbd1. 2026-02-22T19:50:55.607 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../009-librbd1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.607 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.648 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libcephfs2. 2026-02-22T19:50:55.656 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../010-libcephfs2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.657 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.677 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-rados. 2026-02-22T19:50:55.685 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../011-python3-rados_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.685 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.702 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-22T19:50:55.710 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:55.710 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.724 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-cephfs. 2026-02-22T19:50:55.733 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../013-python3-cephfs_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.733 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.748 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-22T19:50:55.756 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../014-python3-ceph-common_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:55.757 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.774 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-22T19:50:55.781 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-22T19:50:55.782 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T19:50:55.797 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-prettytable. 2026-02-22T19:50:55.805 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2026-02-22T19:50:55.806 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-22T19:50:55.820 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-rbd. 2026-02-22T19:50:55.828 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../017-python3-rbd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:55.829 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:55.846 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-22T19:50:55.854 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-22T19:50:55.854 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T19:50:55.873 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libreadline-dev:amd64. 2026-02-22T19:50:55.881 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2026-02-22T19:50:55.881 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T19:50:55.898 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2026-02-22T19:50:55.906 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2026-02-22T19:50:55.907 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T19:50:55.925 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package lua5.1. 2026-02-22T19:50:55.933 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2026-02-22T19:50:55.933 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2026-02-22T19:50:55.954 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package lua-any. 2026-02-22T19:50:55.962 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2026-02-22T19:50:55.963 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking lua-any (27ubuntu1) ... 2026-02-22T19:50:55.976 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package zip. 2026-02-22T19:50:55.984 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2026-02-22T19:50:55.985 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking zip (3.0-12build2) ... 2026-02-22T19:50:56.002 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package unzip. 2026-02-22T19:50:56.010 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2026-02-22T19:50:56.010 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2026-02-22T19:50:56.027 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package luarocks. 2026-02-22T19:50:56.035 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2026-02-22T19:50:56.036 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2026-02-22T19:50:56.073 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package librgw2. 2026-02-22T19:50:56.081 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../026-librgw2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.082 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.153 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-rgw. 2026-02-22T19:50:56.161 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../027-python3-rgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.162 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.177 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-22T19:50:56.185 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../028-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-22T19:50:56.185 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T19:50:56.199 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libradosstriper1. 2026-02-22T19:50:56.207 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../029-libradosstriper1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.208 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.226 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-common. 2026-02-22T19:50:56.234 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../030-ceph-common_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.235 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.436 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-base. 2026-02-22T19:50:56.445 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../031-ceph-base_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:56.450 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:56.512 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-22T19:50:56.521 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-22T19:50:56.521 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-22T19:50:56.536 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-cheroot. 2026-02-22T19:50:56.544 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-22T19:50:56.545 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T19:50:56.562 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-22T19:50:56.570 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-22T19:50:56.571 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-22T19:50:56.585 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-22T19:50:56.594 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-22T19:50:56.594 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-22T19:50:56.608 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-22T19:50:56.616 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-22T19:50:56.616 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-22T19:50:56.630 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-tempora. 2026-02-22T19:50:56.638 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2026-02-22T19:50:56.639 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-22T19:50:56.653 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-portend. 2026-02-22T19:50:56.662 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2026-02-22T19:50:56.662 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-22T19:50:56.676 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-22T19:50:56.684 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-22T19:50:56.684 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-22T19:50:56.699 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-22T19:50:56.707 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-22T19:50:56.707 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-22T19:50:56.730 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-natsort. 2026-02-22T19:50:56.738 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2026-02-22T19:50:56.739 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-22T19:50:56.754 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-logutils. 2026-02-22T19:50:56.762 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2026-02-22T19:50:56.763 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-logutils (0.3.3-8) ... 2026-02-22T19:50:56.778 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-mako. 2026-02-22T19:50:56.786 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2026-02-22T19:50:56.786 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T19:50:56.803 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-simplegeneric. 2026-02-22T19:50:56.810 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2026-02-22T19:50:56.811 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2026-02-22T19:50:56.824 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-singledispatch. 2026-02-22T19:50:56.832 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2026-02-22T19:50:56.833 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2026-02-22T19:50:56.846 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-webob. 2026-02-22T19:50:56.854 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-22T19:50:56.854 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T19:50:56.871 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-waitress. 2026-02-22T19:50:56.879 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1.1_all.deb ... 2026-02-22T19:50:56.881 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T19:50:56.897 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-tempita. 2026-02-22T19:50:56.905 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2026-02-22T19:50:56.905 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T19:50:56.920 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-paste. 2026-02-22T19:50:56.928 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2026-02-22T19:50:56.928 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T19:50:56.956 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2026-02-22T19:50:56.963 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2026-02-22T19:50:56.964 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T19:50:56.978 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-pastedeploy. 2026-02-22T19:50:56.986 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2026-02-22T19:50:56.986 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2026-02-22T19:50:57.002 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-webtest. 2026-02-22T19:50:57.011 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2026-02-22T19:50:57.011 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-webtest (2.0.35-1) ... 2026-02-22T19:50:57.027 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-pecan. 2026-02-22T19:50:57.036 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2026-02-22T19:50:57.036 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T19:50:57.060 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-werkzeug. 2026-02-22T19:50:57.068 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.3_all.deb ... 2026-02-22T19:50:57.069 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T19:50:57.089 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-22T19:50:57.097 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:57.097 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.125 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-22T19:50:57.134 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.134 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.149 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-mgr. 2026-02-22T19:50:57.157 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../057-ceph-mgr_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.157 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.180 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-mon. 2026-02-22T19:50:57.188 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../058-ceph-mon_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.189 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.245 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-22T19:50:57.254 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../059-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-22T19:50:57.254 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T19:50:57.270 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-osd. 2026-02-22T19:50:57.278 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../060-ceph-osd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.278 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.430 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph. 2026-02-22T19:50:57.439 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../061-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.439 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.453 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-fuse. 2026-02-22T19:50:57.461 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../062-ceph-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.461 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.484 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-mds. 2026-02-22T19:50:57.493 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../063-ceph-mds_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.493 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.526 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package cephadm. 2026-02-22T19:50:57.534 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../064-cephadm_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:57.534 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.550 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-22T19:50:57.558 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T19:50:57.559 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:57.579 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-22T19:50:57.587 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:57.587 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.607 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-22T19:50:57.615 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../067-python3-repoze.lru_0.7-2_all.deb ... 2026-02-22T19:50:57.615 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-22T19:50:57.630 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-routes. 2026-02-22T19:50:57.638 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../068-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-22T19:50:57.639 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T19:50:57.658 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-22T19:50:57.666 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../069-ceph-mgr-dashboard_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:57.666 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:57.870 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-22T19:50:57.878 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../070-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-22T19:50:57.879 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T19:50:57.917 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-joblib. 2026-02-22T19:50:57.925 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../071-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-22T19:50:57.925 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T19:50:57.952 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-22T19:50:57.961 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../072-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-22T19:50:57.961 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-22T19:50:57.977 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-sklearn. 2026-02-22T19:50:57.985 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../073-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-22T19:50:57.985 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T19:50:58.069 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-22T19:50:58.077 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../074-ceph-mgr-diskprediction-local_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:58.078 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:58.243 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-cachetools. 2026-02-22T19:50:58.252 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../075-python3-cachetools_5.0.0-1_all.deb ... 2026-02-22T19:50:58.252 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-22T19:50:58.267 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-rsa. 2026-02-22T19:50:58.275 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../076-python3-rsa_4.8-1_all.deb ... 2026-02-22T19:50:58.276 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-22T19:50:58.292 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-google-auth. 2026-02-22T19:50:58.301 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-3_all.deb ... 2026-02-22T19:50:58.301 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-22T19:50:58.318 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-22T19:50:58.326 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-22T19:50:58.326 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T19:50:58.341 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-websocket. 2026-02-22T19:50:58.349 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../079-python3-websocket_1.2.3-1_all.deb ... 2026-02-22T19:50:58.350 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-22T19:50:58.367 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-22T19:50:58.375 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../080-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-22T19:50:58.376 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T19:50:58.468 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-22T19:50:58.478 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:58.478 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:58.493 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-22T19:50:58.500 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../082-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-22T19:50:58.508 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T19:50:58.524 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-22T19:50:58.533 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../083-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T19:50:58.534 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T19:50:58.548 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package jq. 2026-02-22T19:50:58.557 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../084-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T19:50:58.557 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-22T19:50:58.572 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package socat. 2026-02-22T19:50:58.581 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../085-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-22T19:50:58.581 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-22T19:50:58.600 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package xmlstarlet. 2026-02-22T19:50:58.608 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-22T19:50:58.609 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-22T19:50:58.645 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-test. 2026-02-22T19:50:58.653 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../087-ceph-test_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:58.653 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:59.120 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package ceph-volume. 2026-02-22T19:50:59.129 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../088-ceph-volume_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T19:50:59.129 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:59.152 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-22T19:50:59.161 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../089-libcephfs-dev_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:59.161 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:59.175 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package lua-socket:amd64. 2026-02-22T19:50:59.184 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../090-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2026-02-22T19:50:59.184 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T19:50:59.205 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package lua-sec:amd64. 2026-02-22T19:50:59.214 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../091-lua-sec_1.0.2-1_amd64.deb ... 2026-02-22T19:50:59.214 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2026-02-22T19:50:59.231 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package nvme-cli. 2026-02-22T19:50:59.239 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../092-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-22T19:50:59.240 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T19:50:59.273 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package pkg-config. 2026-02-22T19:50:59.282 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../093-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2026-02-22T19:50:59.282 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T19:50:59.298 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-22T19:50:59.306 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T19:50:59.307 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T19:50:59.336 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-22T19:50:59.345 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-22T19:50:59.345 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-22T19:50:59.360 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-pastescript. 2026-02-22T19:50:59.368 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2026-02-22T19:50:59.369 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2026-02-22T19:50:59.386 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-pluggy. 2026-02-22T19:50:59.395 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-22T19:50:59.396 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-22T19:50:59.411 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-psutil. 2026-02-22T19:50:59.420 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-22T19:50:59.420 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-22T19:50:59.439 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-py. 2026-02-22T19:50:59.447 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2026-02-22T19:50:59.448 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-22T19:50:59.468 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-pygments. 2026-02-22T19:50:59.476 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-22T19:50:59.476 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T19:50:59.517 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-pyinotify. 2026-02-22T19:50:59.526 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2026-02-22T19:50:59.526 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2026-02-22T19:50:59.542 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-toml. 2026-02-22T19:50:59.551 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2026-02-22T19:50:59.551 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-22T19:50:59.566 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-pytest. 2026-02-22T19:50:59.574 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-22T19:50:59.575 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T19:50:59.596 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package python3-simplejson. 2026-02-22T19:50:59.604 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-22T19:50:59.605 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-22T19:50:59.623 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-22T19:50:59.632 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-22T19:50:59.632 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-22T19:50:59.700 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package radosgw. 2026-02-22T19:50:59.709 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../106-radosgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:59.709 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:59.821 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package rbd-fuse. 2026-02-22T19:50:59.830 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../107-rbd-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:59.830 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:59.846 INFO:teuthology.orchestra.run.trial007.stdout:Selecting previously unselected package rbd-mirror. 2026-02-22T19:50:59.854 INFO:teuthology.orchestra.run.trial007.stdout:Preparing to unpack .../108-rbd-mirror_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T19:50:59.854 INFO:teuthology.orchestra.run.trial007.stdout:Unpacking rbd-mirror (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:50:59.915 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-22T19:50:59.983 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T19:50:59.984 INFO:teuthology.orchestra.run.trial007.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T19:51:00.048 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T19:51:00.339 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-22T19:51:00.725 INFO:teuthology.orchestra.run.trial007.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-22T19:51:00.735 INFO:teuthology.orchestra.run.trial007.stdout:Setting up cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:00.779 INFO:teuthology.orchestra.run.trial007.stdout:Adding system user cephadm....done 2026-02-22T19:51:00.790 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T19:51:00.862 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-22T19:51:00.930 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T19:51:00.931 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-22T19:51:01.005 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-22T19:51:01.080 INFO:teuthology.orchestra.run.trial007.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T19:51:01.081 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-22T19:51:01.177 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T19:51:01.299 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-22T19:51:01.374 INFO:teuthology.orchestra.run.trial007.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2026-02-22T19:51:01.384 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2026-02-22T19:51:01.457 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-22T19:51:01.531 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:01.599 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T19:51:01.600 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-22T19:51:01.601 INFO:teuthology.orchestra.run.trial007.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T19:51:01.601 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T19:51:01.602 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T19:51:01.603 INFO:teuthology.orchestra.run.trial007.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2026-02-22T19:51:01.620 INFO:teuthology.orchestra.run.trial007.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2026-02-22T19:51:01.622 INFO:teuthology.orchestra.run.trial007.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2026-02-22T19:51:01.623 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T19:51:01.624 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-22T19:51:01.746 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-22T19:51:01.826 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T19:51:01.905 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-22T19:51:01.992 INFO:teuthology.orchestra.run.trial007.stdout:Setting up zip (3.0-12build2) ... 2026-02-22T19:51:01.993 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T19:51:02.244 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T19:51:02.321 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T19:51:02.322 INFO:teuthology.orchestra.run.trial007.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-22T19:51:02.323 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T19:51:02.419 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T19:51:02.539 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T19:51:02.668 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T19:51:02.751 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T19:51:02.856 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-22T19:51:02.933 INFO:teuthology.orchestra.run.trial007.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-22T19:51:02.934 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:03.015 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T19:51:03.512 INFO:teuthology.orchestra.run.trial007.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T19:51:03.533 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:51:03.538 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-22T19:51:03.616 INFO:teuthology.orchestra.run.trial007.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T19:51:03.617 INFO:teuthology.orchestra.run.trial007.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-22T19:51:03.618 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-22T19:51:03.692 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-22T19:51:03.764 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:51:03.766 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-22T19:51:03.842 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2026-02-22T19:51:03.917 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-logutils (0.3.3-8) ... 2026-02-22T19:51:03.993 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-22T19:51:04.070 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2026-02-22T19:51:04.140 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-22T19:51:04.220 INFO:teuthology.orchestra.run.trial007.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T19:51:04.221 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-22T19:51:04.304 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T19:51:04.305 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T19:51:04.381 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T19:51:04.459 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T19:51:04.543 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-22T19:51:04.618 INFO:teuthology.orchestra.run.trial007.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T19:51:04.619 INFO:teuthology.orchestra.run.trial007.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2026-02-22T19:51:04.620 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T19:51:04.620 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T19:51:04.754 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2026-02-22T19:51:04.832 INFO:teuthology.orchestra.run.trial007.stdout:Setting up lua-any (27ubuntu1) ... 2026-02-22T19:51:04.833 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-22T19:51:04.908 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T19:51:04.909 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-22T19:51:04.993 INFO:teuthology.orchestra.run.trial007.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-22T19:51:04.994 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-webtest (2.0.35-1) ... 2026-02-22T19:51:05.075 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-22T19:51:05.205 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-pastescript (2.0.2-4) ... 2026-02-22T19:51:05.293 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T19:51:05.405 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T19:51:05.406 INFO:teuthology.orchestra.run.trial007.stdout:Setting up librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.407 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.408 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T19:51:05.924 INFO:teuthology.orchestra.run.trial007.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2026-02-22T19:51:05.935 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.936 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.937 INFO:teuthology.orchestra.run.trial007.stdout:Setting up librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.938 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:05.939 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.011 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T19:51:06.011 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T19:51:06.412 INFO:teuthology.orchestra.run.trial007.stdout:Setting up libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.413 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.414 INFO:teuthology.orchestra.run.trial007.stdout:Setting up librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.415 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.416 INFO:teuthology.orchestra.run.trial007.stdout:Setting up rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.417 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.417 INFO:teuthology.orchestra.run.trial007.stdout:Setting up python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.418 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:06.445 INFO:teuthology.orchestra.run.trial007.stdout:Adding group ceph....done 2026-02-22T19:51:06.482 INFO:teuthology.orchestra.run.trial007.stdout:Adding system user ceph....done 2026-02-22T19:51:06.491 INFO:teuthology.orchestra.run.trial007.stdout:Setting system user ceph properties....done 2026-02-22T19:51:06.498 INFO:teuthology.orchestra.run.trial007.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-22T19:51:06.586 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-22T19:51:06.851 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-22T19:51:07.254 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:07.255 INFO:teuthology.orchestra.run.trial007.stdout:Setting up radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:07.519 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T19:51:07.520 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T19:51:07.906 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:08.010 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-22T19:51:08.403 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:08.462 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T19:51:08.463 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T19:51:08.866 INFO:teuthology.orchestra.run.trial007.stdout:Setting up rbd-mirror (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:08.942 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T19:51:08.943 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T19:51:09.334 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:09.411 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T19:51:09.411 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T19:51:09.786 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:09.883 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T19:51:09.883 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T19:51:10.273 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:10.273 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:10.283 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:10.357 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T19:51:10.357 INFO:teuthology.orchestra.run.trial007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T19:51:10.747 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:10.756 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:10.757 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:10.767 INFO:teuthology.orchestra.run.trial007.stdout:Setting up ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T19:51:10.876 INFO:teuthology.orchestra.run.trial007.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2026-02-22T19:51:10.883 INFO:teuthology.orchestra.run.trial007.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-22T19:51:11.124 INFO:teuthology.orchestra.run.trial007.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-22T19:51:11.207 INFO:teuthology.orchestra.run.trial007.stdout:Processing triggers for install-info (6.8-4build1) ... 2026-02-22T19:51:11.695 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:11.695 INFO:teuthology.orchestra.run.trial007.stdout:Running kernel seems to be up-to-date. 2026-02-22T19:51:11.695 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:11.695 INFO:teuthology.orchestra.run.trial007.stdout:The processor microcode seems to be up-to-date. 2026-02-22T19:51:11.696 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:11.696 INFO:teuthology.orchestra.run.trial007.stdout:No services need to be restarted. 2026-02-22T19:51:11.696 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:11.696 INFO:teuthology.orchestra.run.trial007.stdout:No containers need to be restarted. 2026-02-22T19:51:11.696 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:11.696 INFO:teuthology.orchestra.run.trial007.stdout:No user sessions are running outdated binaries. 2026-02-22T19:51:11.696 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:11.696 INFO:teuthology.orchestra.run.trial007.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-22T19:51:12.491 INFO:teuthology.orchestra.run.trial007.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-22T19:51:12.498 DEBUG:teuthology.parallel:result is None 2026-02-22T19:51:12.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:51:12.669 DEBUG:teuthology.orchestra.run.trial007:> dpkg-query -W -f '${Version}' ceph 2026-02-22T19:51:12.680 INFO:teuthology.orchestra.run.trial007.stdout:19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:51:12.681 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:51:12.681 INFO:teuthology.task.install:The correct ceph version 19.2.3-689-gf90f4eae-1jammy is installed. 2026-02-22T19:51:12.684 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:51:12.841 DEBUG:teuthology.orchestra.run.trial017:> dpkg-query -W -f '${Version}' ceph 2026-02-22T19:51:12.852 INFO:teuthology.orchestra.run.trial017.stdout:19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:51:12.852 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T19:51:12.852 INFO:teuthology.task.install:The correct ceph version 19.2.3-689-gf90f4eae-1jammy is installed. 2026-02-22T19:51:12.854 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-22T19:51:12.854 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:12.854 DEBUG:teuthology.orchestra.run.trial007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T19:51:12.865 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:12.865 DEBUG:teuthology.orchestra.run.trial017:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T19:51:12.903 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-22T19:51:12.903 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:12.903 DEBUG:teuthology.orchestra.run.trial007:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T19:51:12.919 DEBUG:teuthology.orchestra.run.trial007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T19:51:12.967 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:12.968 DEBUG:teuthology.orchestra.run.trial017:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T19:51:12.976 DEBUG:teuthology.orchestra.run.trial017:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T19:51:13.026 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-22T19:51:13.026 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:13.026 DEBUG:teuthology.orchestra.run.trial007:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T19:51:13.035 DEBUG:teuthology.orchestra.run.trial007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T19:51:13.083 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:13.083 DEBUG:teuthology.orchestra.run.trial017:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T19:51:13.095 DEBUG:teuthology.orchestra.run.trial017:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T19:51:13.146 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-22T19:51:13.147 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:13.147 DEBUG:teuthology.orchestra.run.trial007:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T19:51:13.156 DEBUG:teuthology.orchestra.run.trial007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T19:51:13.207 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:13.207 DEBUG:teuthology.orchestra.run.trial017:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T19:51:13.216 DEBUG:teuthology.orchestra.run.trial017:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T19:51:13.266 INFO:teuthology.run_tasks:Running task ceph... 2026-02-22T19:51:13.380 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-22T19:51:13.380 DEBUG:teuthology.orchestra.run.trial007:> sudo chmod 777 /var/log/ceph 2026-02-22T19:51:13.381 DEBUG:teuthology.orchestra.run.trial017:> sudo chmod 777 /var/log/ceph 2026-02-22T19:51:13.389 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-22T19:51:13.389 DEBUG:teuthology.orchestra.run.trial007:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-22T19:51:13.432 DEBUG:teuthology.orchestra.run.trial017:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-22T19:51:13.440 INFO:tasks.ceph:Creating extra log directories... 2026-02-22T19:51:13.440 DEBUG:teuthology.orchestra.run.trial007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-22T19:51:13.484 DEBUG:teuthology.orchestra.run.trial017:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-22T19:51:13.513 INFO:tasks.ceph:Creating ceph cluster cluster1... 2026-02-22T19:51:13.513 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-22T19:51:13.514 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974', 'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-journaling}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '63974', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror.sh workunit to test the rbd-mirror daemon in snapshot mode'}], 'name': 'yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9196, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'targets': {'trial007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJMUlU4Cteug0GiCx5j2Uet2DqUnnKtQrIB7U1GQif+JQTZksVOpHfZDaz1UwK4wK9qYTa3nRdx4FZj5aIdfkyo=', 'trial017.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLDW5vyj+GviZNpv6neyV3hYsSiL3/fKOpVwfikmufzMXNVE4sj5SIJFR4lUNFi/6pYMJ2OdRg0h3yjhc8eG8CM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,journaling', 'RBD_MIRROR_CONFIG_KEY': '1', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-22_19:35:38', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-22T19:51:13.515 DEBUG:teuthology.orchestra.run.trial007:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2026-02-22T19:51:13.537 DEBUG:teuthology.orchestra.run.trial017:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2026-02-22T19:51:13.542 DEBUG:teuthology.orchestra.run.trial007:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T19:51:13.589 DEBUG:teuthology.orchestra.run.trial017:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T19:51:13.599 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:13.599 DEBUG:teuthology.orchestra.run.trial007:> dd if=/scratch_devs of=/dev/stdout 2026-02-22T19:51:13.644 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-22T19:51:13.645 DEBUG:teuthology.orchestra.run.trial007:> stat /dev/vg_nvme/lv_1 2026-02-22T19:51:13.696 INFO:teuthology.orchestra.run.trial007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-22T19:51:13.696 INFO:teuthology.orchestra.run.trial007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:13.696 INFO:teuthology.orchestra.run.trial007.stdout:Device: 5h/5d Inode: 918 Links: 1 2026-02-22T19:51:13.696 INFO:teuthology.orchestra.run.trial007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:13.696 INFO:teuthology.orchestra.run.trial007.stdout:Access: 2026-02-22 19:46:38.779186066 +0000 2026-02-22T19:51:13.696 INFO:teuthology.orchestra.run.trial007.stdout:Modify: 2026-02-22 19:46:38.775186135 +0000 2026-02-22T19:51:13.696 INFO:teuthology.orchestra.run.trial007.stdout:Change: 2026-02-22 19:46:38.775186135 +0000 2026-02-22T19:51:13.696 INFO:teuthology.orchestra.run.trial007.stdout: Birth: - 2026-02-22T19:51:13.697 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-22T19:51:13.750 INFO:teuthology.orchestra.run.trial007.stderr:1+0 records in 2026-02-22T19:51:13.751 INFO:teuthology.orchestra.run.trial007.stderr:1+0 records out 2026-02-22T19:51:13.751 INFO:teuthology.orchestra.run.trial007.stderr:512 bytes copied, 0.000177418 s, 2.9 MB/s 2026-02-22T19:51:13.752 DEBUG:teuthology.orchestra.run.trial007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-22T19:51:13.801 DEBUG:teuthology.orchestra.run.trial007:> stat /dev/vg_nvme/lv_2 2026-02-22T19:51:13.852 INFO:teuthology.orchestra.run.trial007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-22T19:51:13.852 INFO:teuthology.orchestra.run.trial007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:13.852 INFO:teuthology.orchestra.run.trial007.stdout:Device: 5h/5d Inode: 929 Links: 1 2026-02-22T19:51:13.852 INFO:teuthology.orchestra.run.trial007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:13.852 INFO:teuthology.orchestra.run.trial007.stdout:Access: 2026-02-22 19:46:39.095180693 +0000 2026-02-22T19:51:13.852 INFO:teuthology.orchestra.run.trial007.stdout:Modify: 2026-02-22 19:46:39.091180762 +0000 2026-02-22T19:51:13.852 INFO:teuthology.orchestra.run.trial007.stdout:Change: 2026-02-22 19:46:39.091180762 +0000 2026-02-22T19:51:13.852 INFO:teuthology.orchestra.run.trial007.stdout: Birth: - 2026-02-22T19:51:13.853 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-22T19:51:13.903 INFO:teuthology.orchestra.run.trial007.stderr:1+0 records in 2026-02-22T19:51:13.903 INFO:teuthology.orchestra.run.trial007.stderr:1+0 records out 2026-02-22T19:51:13.903 INFO:teuthology.orchestra.run.trial007.stderr:512 bytes copied, 0.000178951 s, 2.9 MB/s 2026-02-22T19:51:13.904 DEBUG:teuthology.orchestra.run.trial007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-22T19:51:13.957 DEBUG:teuthology.orchestra.run.trial007:> stat /dev/vg_nvme/lv_3 2026-02-22T19:51:14.004 INFO:teuthology.orchestra.run.trial007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-22T19:51:14.004 INFO:teuthology.orchestra.run.trial007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:14.005 INFO:teuthology.orchestra.run.trial007.stdout:Device: 5h/5d Inode: 965 Links: 1 2026-02-22T19:51:14.005 INFO:teuthology.orchestra.run.trial007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:14.005 INFO:teuthology.orchestra.run.trial007.stdout:Access: 2026-02-22 19:46:39.415175254 +0000 2026-02-22T19:51:14.005 INFO:teuthology.orchestra.run.trial007.stdout:Modify: 2026-02-22 19:46:39.411175322 +0000 2026-02-22T19:51:14.005 INFO:teuthology.orchestra.run.trial007.stdout:Change: 2026-02-22 19:46:39.411175322 +0000 2026-02-22T19:51:14.005 INFO:teuthology.orchestra.run.trial007.stdout: Birth: - 2026-02-22T19:51:14.006 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-22T19:51:14.059 INFO:teuthology.orchestra.run.trial007.stderr:1+0 records in 2026-02-22T19:51:14.059 INFO:teuthology.orchestra.run.trial007.stderr:1+0 records out 2026-02-22T19:51:14.059 INFO:teuthology.orchestra.run.trial007.stderr:512 bytes copied, 0.000187738 s, 2.7 MB/s 2026-02-22T19:51:14.060 DEBUG:teuthology.orchestra.run.trial007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-22T19:51:14.113 DEBUG:teuthology.orchestra.run.trial007:> stat /dev/vg_nvme/lv_4 2026-02-22T19:51:14.159 INFO:teuthology.orchestra.run.trial007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-22T19:51:14.160 INFO:teuthology.orchestra.run.trial007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:14.160 INFO:teuthology.orchestra.run.trial007.stdout:Device: 5h/5d Inode: 977 Links: 1 2026-02-22T19:51:14.160 INFO:teuthology.orchestra.run.trial007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:14.160 INFO:teuthology.orchestra.run.trial007.stdout:Access: 2026-02-22 19:46:39.883167296 +0000 2026-02-22T19:51:14.160 INFO:teuthology.orchestra.run.trial007.stdout:Modify: 2026-02-22 19:46:39.727169949 +0000 2026-02-22T19:51:14.160 INFO:teuthology.orchestra.run.trial007.stdout:Change: 2026-02-22 19:46:39.727169949 +0000 2026-02-22T19:51:14.160 INFO:teuthology.orchestra.run.trial007.stdout: Birth: - 2026-02-22T19:51:14.161 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-22T19:51:14.214 INFO:teuthology.orchestra.run.trial007.stderr:1+0 records in 2026-02-22T19:51:14.215 INFO:teuthology.orchestra.run.trial007.stderr:1+0 records out 2026-02-22T19:51:14.215 INFO:teuthology.orchestra.run.trial007.stderr:512 bytes copied, 0.000159355 s, 3.2 MB/s 2026-02-22T19:51:14.216 DEBUG:teuthology.orchestra.run.trial007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-22T19:51:14.265 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:14.265 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial007.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T19:51:14.265 INFO:tasks.ceph:Generating config... 2026-02-22T19:51:14.267 INFO:tasks.ceph:[client] debug rbd = 20 2026-02-22T19:51:14.268 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2026-02-22T19:51:14.268 INFO:tasks.ceph:[client] log to stderr = False 2026-02-22T19:51:14.268 INFO:tasks.ceph:[client] rbd default features = 125 2026-02-22T19:51:14.268 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.268 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.268 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.268 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.268 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.268 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.269 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.269 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.269 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.269 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.269 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.269 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.269 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.269 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.269 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-22T19:51:14.269 INFO:tasks.ceph:[global] mon warn on pool no app = False 2026-02-22T19:51:14.269 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2026-02-22T19:51:14.270 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-22T19:51:14.270 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-22T19:51:14.270 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-22T19:51:14.270 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-22T19:51:14.270 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-22T19:51:14.270 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-22T19:51:14.270 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2026-02-22T19:51:14.270 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2026-02-22T19:51:14.270 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-22T19:51:14.270 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2026-02-22T19:51:14.270 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2026-02-22T19:51:14.270 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-22T19:51:14.271 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-22T19:51:14.271 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2026-02-22T19:51:14.271 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:14.271 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-22T19:51:14.271 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-22T19:51:14.271 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-22T19:51:14.271 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-22T19:51:14.271 INFO:tasks.ceph:Setting up cluster1.mon.a... 2026-02-22T19:51:14.271 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2026-02-22T19:51:14.323 INFO:teuthology.orchestra.run.trial007.stdout:creating /etc/ceph/cluster1.keyring 2026-02-22T19:51:14.326 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2026-02-22T19:51:14.386 DEBUG:teuthology.orchestra.run.trial007:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-22T19:51:14.435 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '10.20.193.7')] 2026-02-22T19:51:14.436 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.7', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2026-02-22T19:51:14.437 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:14.438 DEBUG:teuthology.orchestra.run.trial007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T19:51:14.484 DEBUG:teuthology.orchestra.run.trial007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.7 --print /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:14.547 INFO:teuthology.orchestra.run.trial007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:14.548 INFO:teuthology.orchestra.run.trial007.stdout:monmaptool: generated fsid a0d80545-8670-4b47-b26d-3047f07e2d2b 2026-02-22T19:51:14.548 INFO:teuthology.orchestra.run.trial007.stdout:setting min_mon_release = quincy 2026-02-22T19:51:14.548 INFO:teuthology.orchestra.run.trial007.stdout:epoch 0 2026-02-22T19:51:14.548 INFO:teuthology.orchestra.run.trial007.stdout:fsid a0d80545-8670-4b47-b26d-3047f07e2d2b 2026-02-22T19:51:14.548 INFO:teuthology.orchestra.run.trial007.stdout:last_changed 2026-02-22T19:51:14.545986+0000 2026-02-22T19:51:14.548 INFO:teuthology.orchestra.run.trial007.stdout:created 2026-02-22T19:51:14.545986+0000 2026-02-22T19:51:14.548 INFO:teuthology.orchestra.run.trial007.stdout:min_mon_release 17 (quincy) 2026-02-22T19:51:14.548 INFO:teuthology.orchestra.run.trial007.stdout:election_strategy: 1 2026-02-22T19:51:14.549 INFO:teuthology.orchestra.run.trial007.stdout:0: [v2:10.20.193.7:3300/0,v1:10.20.193.7:6789/0] mon.a 2026-02-22T19:51:14.549 INFO:teuthology.orchestra.run.trial007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2026-02-22T19:51:14.550 DEBUG:teuthology.orchestra.run.trial007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T19:51:14.599 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID a0d80545-8670-4b47-b26d-3047f07e2d2b... 2026-02-22T19:51:14.600 DEBUG:teuthology.orchestra.run.trial007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2026-02-22T19:51:14.645 DEBUG:teuthology.orchestra.run.trial017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2026-02-22T19:51:14.664 INFO:teuthology.orchestra.run.trial007.stdout:[global] 2026-02-22T19:51:14.664 INFO:teuthology.orchestra.run.trial007.stdout: chdir = "" 2026-02-22T19:51:14.665 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T19:51:14.665 INFO:teuthology.orchestra.run.trial007.stdout: auth supported = cephx 2026-02-22T19:51:14.665 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.665 INFO:teuthology.orchestra.run.trial007.stdout: filestore xattr use omap = true 2026-02-22T19:51:14.665 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.665 INFO:teuthology.orchestra.run.trial007.stdout: mon clock drift allowed = 1.000 2026-02-22T19:51:14.665 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.665 INFO:teuthology.orchestra.run.trial007.stdout: osd crush chooseleaf type = 0 2026-02-22T19:51:14.665 INFO:teuthology.orchestra.run.trial007.stdout: auth debug = true 2026-02-22T19:51:14.665 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: ms die on old message = true 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: ms die on bug = true 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: mon pg warn max object skew = 0 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: osd pool default size = 2 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.666 INFO:teuthology.orchestra.run.trial007.stdout: mon osd allow primary affinity = true 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: mon osd allow pg remap = true 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on legacy crush tunables = false 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on crush straw calc version zero = false 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on no sortbitwise = false 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on osd down out interval zero = false 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on too few osds = false 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: mon_allow_pool_size_one = true 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.667 INFO:teuthology.orchestra.run.trial007.stdout: osd default data pool replay window = 5 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: mon allow pool delete = true 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: mon cluster log file level = debug 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: debug asserts on shutdown = true 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: mon health detail to clog = false 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: mon host = 10.20.193.7 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: mon client directed command retry = 5 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on pool no app = False 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: ms inject socket failures = 5000 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: fsid = a0d80545-8670-4b47-b26d-3047f07e2d2b 2026-02-22T19:51:14.668 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout:[osd] 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout: osd journal size = 100 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout: osd scrub load threshold = 5.0 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout: osd scrub max interval = 600 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout: osd mclock profile = high_recovery_ops 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout: osd mclock skip benchmark = true 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout: osd recover clone overlap = true 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout: osd recovery max chunk = 1048576 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.669 INFO:teuthology.orchestra.run.trial007.stdout: osd debug shutdown = true 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: osd debug op order = true 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: osd debug verify stray on activate = true 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: osd debug trim objects = true 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: osd open classes on start = true 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: osd debug pg log writeout = true 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: osd deep scrub update digest min age = 30 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: osd map max advance = 10 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: journal zero on create = true 2026-02-22T19:51:14.670 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: filestore ondisk finisher threads = 3 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: filestore apply finisher threads = 3 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: bdev debug aio = true 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: osd debug misdirected ops = true 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: bluestore block size = 96636764160 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: bluestore compression algorithm = zlib 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: bluestore compression mode = aggressive 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: bluestore fsck on mount = True 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: debug bluefs = 1/20 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: debug bluestore = 1/20 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: debug ms = 1 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: debug osd = 20 2026-02-22T19:51:14.671 INFO:teuthology.orchestra.run.trial007.stdout: debug rocksdb = 4/10 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout: mon osd full ratio = 0.9 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout: mon osd nearfull ratio = 0.8 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout: osd failsafe full ratio = 0.95 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout: osd objectstore = bluestore 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout:[mgr] 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout: debug ms = 1 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout: debug mgr = 20 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout: debug mon = 20 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout: debug auth = 20 2026-02-22T19:51:14.672 INFO:teuthology.orchestra.run.trial007.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: mgr/telemetry/nag = false 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout:[mon] 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: debug ms = 1 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: debug mon = 20 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: debug paxos = 20 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: debug auth = 20 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: mon data avail warn = 5 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: mon mgr mkfs grace = 240 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: mon osd reporter subtree level = osd 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: mon osd prime pg temp = true 2026-02-22T19:51:14.673 INFO:teuthology.orchestra.run.trial007.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: # 1m isn't quite enough 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: mon_down_mkfs_grace = 2m 2026-02-22T19:51:14.674 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout: mon_warn_on_filestore_osds = false 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout:[client] 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout: rgw cache enabled = true 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout: rgw enable ops log = true 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout: rgw enable usage log = true 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout: debug rbd = 20 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout: debug rbd_mirror = 15 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout: log to stderr = False 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout: rbd default features = 125 2026-02-22T19:51:14.675 INFO:teuthology.orchestra.run.trial007.stdout:[mon.a] 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.0] 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.1] 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.2] 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.3] 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.676 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.4] 2026-02-22T19:51:14.677 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.677 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.677 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.5] 2026-02-22T19:51:14.677 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.677 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.677 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.6] 2026-02-22T19:51:14.677 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.677 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.679 INFO:teuthology.orchestra.run.trial017.stdout:[global] 2026-02-22T19:51:14.679 INFO:teuthology.orchestra.run.trial017.stdout: chdir = "" 2026-02-22T19:51:14.679 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T19:51:14.679 INFO:teuthology.orchestra.run.trial017.stdout: auth supported = cephx 2026-02-22T19:51:14.679 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.679 INFO:teuthology.orchestra.run.trial017.stdout: filestore xattr use omap = true 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: mon clock drift allowed = 1.000 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: osd crush chooseleaf type = 0 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: auth debug = true 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: ms die on old message = true 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: ms die on bug = true 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: mon pg warn max object skew = 0 2026-02-22T19:51:14.680 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: osd pool default size = 2 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: mon osd allow primary affinity = true 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: mon osd allow pg remap = true 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on legacy crush tunables = false 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on crush straw calc version zero = false 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on no sortbitwise = false 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on osd down out interval zero = false 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on too few osds = false 2026-02-22T19:51:14.681 INFO:teuthology.orchestra.run.trial017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: mon_allow_pool_size_one = true 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: osd default data pool replay window = 5 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: mon allow pool delete = true 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: mon cluster log file level = debug 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: debug asserts on shutdown = true 2026-02-22T19:51:14.682 INFO:teuthology.orchestra.run.trial017.stdout: mon health detail to clog = false 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout: mon host = 10.20.193.7 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout: mon client directed command retry = 5 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on pool no app = False 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout: ms inject socket failures = 5000 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout: fsid = a0d80545-8670-4b47-b26d-3047f07e2d2b 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout:[osd] 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout: osd journal size = 100 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout: osd scrub load threshold = 5.0 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout: osd scrub max interval = 600 2026-02-22T19:51:14.683 INFO:teuthology.orchestra.run.trial017.stdout: osd mclock profile = high_recovery_ops 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: osd mclock skip benchmark = true 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: osd recover clone overlap = true 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: osd recovery max chunk = 1048576 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: osd debug shutdown = true 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: osd debug op order = true 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: osd debug verify stray on activate = true 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: osd debug trim objects = true 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: osd open classes on start = true 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: osd debug pg log writeout = true 2026-02-22T19:51:14.684 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: osd deep scrub update digest min age = 30 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: osd map max advance = 10 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: journal zero on create = true 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: filestore ondisk finisher threads = 3 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: filestore apply finisher threads = 3 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: bdev debug aio = true 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: osd debug misdirected ops = true 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: bluestore block size = 96636764160 2026-02-22T19:51:14.685 INFO:teuthology.orchestra.run.trial017.stdout: bluestore compression algorithm = zlib 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: bluestore compression mode = aggressive 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: bluestore fsck on mount = True 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: debug bluefs = 1/20 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: debug bluestore = 1/20 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: debug ms = 1 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: debug osd = 20 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: debug rocksdb = 4/10 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: mon osd full ratio = 0.9 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: mon osd nearfull ratio = 0.8 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: osd failsafe full ratio = 0.95 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: osd objectstore = bluestore 2026-02-22T19:51:14.686 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout:[mgr] 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout: debug ms = 1 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout: debug mgr = 20 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout: debug mon = 20 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout: debug auth = 20 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout: mgr/telemetry/nag = false 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout:[mon] 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout: debug ms = 1 2026-02-22T19:51:14.687 INFO:teuthology.orchestra.run.trial017.stdout: debug mon = 20 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: debug paxos = 20 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: debug auth = 20 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: mon data avail warn = 5 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: mon mgr mkfs grace = 240 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: mon osd reporter subtree level = osd 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: mon osd prime pg temp = true 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T19:51:14.688 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout: # 1m isn't quite enough 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout: mon_down_mkfs_grace = 2m 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout: mon_warn_on_filestore_osds = false 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout:[client] 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout: rgw cache enabled = true 2026-02-22T19:51:14.689 INFO:teuthology.orchestra.run.trial017.stdout: rgw enable ops log = true 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout: rgw enable usage log = true 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout: debug rbd = 20 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout: debug rbd_mirror = 15 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout: log to stderr = False 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout: rbd default features = 125 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout:[mon.a] 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.0] 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.1] 2026-02-22T19:51:14.690 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.2] 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.3] 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.4] 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.5] 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.691 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.692 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.6] 2026-02-22T19:51:14.692 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:14.692 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:14.692 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2026-02-22T19:51:14.692 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2026-02-22T19:51:14.745 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-22T19:51:14.746 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:14.746 DEBUG:teuthology.orchestra.run.trial007:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2026-02-22T19:51:14.796 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:14.796 DEBUG:teuthology.orchestra.run.trial007:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2026-02-22T19:51:14.844 INFO:tasks.ceph:Sending monmap to node ubuntu@trial007.front.sepia.ceph.com 2026-02-22T19:51:14.844 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:14.844 DEBUG:teuthology.orchestra.run.trial007:> sudo dd of=/etc/ceph/cluster1.keyring 2026-02-22T19:51:14.844 DEBUG:teuthology.orchestra.run.trial007:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-22T19:51:14.901 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:14.901 DEBUG:teuthology.orchestra.run.trial007:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:14.948 INFO:tasks.ceph:Sending monmap to node ubuntu@trial017.front.sepia.ceph.com 2026-02-22T19:51:14.948 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:14.948 DEBUG:teuthology.orchestra.run.trial017:> sudo dd of=/etc/ceph/cluster1.keyring 2026-02-22T19:51:14.948 DEBUG:teuthology.orchestra.run.trial017:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-22T19:51:14.963 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:14.964 DEBUG:teuthology.orchestra.run.trial017:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:15.010 INFO:tasks.ceph:Setting up mon nodes... 2026-02-22T19:51:15.010 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-22T19:51:15.011 DEBUG:teuthology.orchestra.run.trial007:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2026-02-22T19:51:15.033 INFO:teuthology.orchestra.run.trial007.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2026-02-22T19:51:15.035 INFO:tasks.ceph:Setting up mds nodes... 2026-02-22T19:51:15.035 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-22T19:51:15.036 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2026-02-22T19:51:15.092 INFO:teuthology.orchestra.run.trial007.stdout:creating /etc/ceph/cluster1.client.0.keyring 2026-02-22T19:51:15.099 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2026-02-22T19:51:15.119 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2026-02-22T19:51:15.127 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2026-02-22T19:51:15.193 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2026-02-22T19:51:15.201 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2026-02-22T19:51:15.264 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2026-02-22T19:51:15.274 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2026-02-22T19:51:15.336 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2026-02-22T19:51:15.344 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2026-02-22T19:51:15.412 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2026-02-22T19:51:15.420 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2026-02-22T19:51:15.484 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2026-02-22T19:51:15.492 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2026-02-22T19:51:15.556 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2026-02-22T19:51:15.564 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2026-02-22T19:51:15.628 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2026-02-22T19:51:15.636 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-22T19:51:15.636 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial007.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T19:51:15.636 DEBUG:teuthology.orchestra.run.trial007:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2026-02-22T19:51:15.645 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:15.645 INFO:tasks.ceph:role: cluster1.osd.0 2026-02-22T19:51:15.645 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial007.front.sepia.ceph.com 2026-02-22T19:51:15.646 DEBUG:teuthology.orchestra.run.trial007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-22T19:51:15.696 INFO:teuthology.orchestra.run.trial007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:15.697 INFO:teuthology.orchestra.run.trial007.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:15.697 INFO:teuthology.orchestra.run.trial007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:15.697 INFO:teuthology.orchestra.run.trial007.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:15.697 INFO:teuthology.orchestra.run.trial007.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:15.697 INFO:teuthology.orchestra.run.trial007.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:15.697 INFO:teuthology.orchestra.run.trial007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:15.697 INFO:teuthology.orchestra.run.trial007.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:15.698 INFO:teuthology.orchestra.run.trial007.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:15.698 INFO:teuthology.orchestra.run.trial007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:16.278 INFO:teuthology.orchestra.run.trial007.stdout:Discarding blocks...Done. 2026-02-22T19:51:16.280 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial007.front.sepia.ceph.com -o noatime 2026-02-22T19:51:16.280 DEBUG:teuthology.orchestra.run.trial007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2026-02-22T19:51:16.340 DEBUG:teuthology.orchestra.run.trial007:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2026-02-22T19:51:16.389 INFO:teuthology.orchestra.run.trial007.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:16.390 DEBUG:teuthology.orchestra.run.trial007:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2026-02-22T19:51:16.439 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:16.440 INFO:tasks.ceph:role: cluster1.osd.1 2026-02-22T19:51:16.440 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial007.front.sepia.ceph.com 2026-02-22T19:51:16.440 DEBUG:teuthology.orchestra.run.trial007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-22T19:51:16.497 INFO:teuthology.orchestra.run.trial007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:16.498 INFO:teuthology.orchestra.run.trial007.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:16.498 INFO:teuthology.orchestra.run.trial007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:16.498 INFO:teuthology.orchestra.run.trial007.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:16.498 INFO:teuthology.orchestra.run.trial007.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:16.498 INFO:teuthology.orchestra.run.trial007.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:16.499 INFO:teuthology.orchestra.run.trial007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:16.499 INFO:teuthology.orchestra.run.trial007.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:16.499 INFO:teuthology.orchestra.run.trial007.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:16.499 INFO:teuthology.orchestra.run.trial007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:17.061 INFO:teuthology.orchestra.run.trial007.stdout:Discarding blocks...Done. 2026-02-22T19:51:17.063 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial007.front.sepia.ceph.com -o noatime 2026-02-22T19:51:17.063 DEBUG:teuthology.orchestra.run.trial007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2026-02-22T19:51:17.121 DEBUG:teuthology.orchestra.run.trial007:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2026-02-22T19:51:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:17.171 INFO:teuthology.orchestra.run.trial007.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:17.171 DEBUG:teuthology.orchestra.run.trial007:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2026-02-22T19:51:17.223 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:17.224 INFO:tasks.ceph:role: cluster1.osd.2 2026-02-22T19:51:17.224 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial007.front.sepia.ceph.com 2026-02-22T19:51:17.224 DEBUG:teuthology.orchestra.run.trial007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-22T19:51:17.277 INFO:teuthology.orchestra.run.trial007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:17.277 INFO:teuthology.orchestra.run.trial007.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:17.277 INFO:teuthology.orchestra.run.trial007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:17.277 INFO:teuthology.orchestra.run.trial007.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:17.277 INFO:teuthology.orchestra.run.trial007.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:17.277 INFO:teuthology.orchestra.run.trial007.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:17.277 INFO:teuthology.orchestra.run.trial007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:17.277 INFO:teuthology.orchestra.run.trial007.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:17.277 INFO:teuthology.orchestra.run.trial007.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:17.278 INFO:teuthology.orchestra.run.trial007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:17.844 INFO:teuthology.orchestra.run.trial007.stdout:Discarding blocks...Done. 2026-02-22T19:51:17.845 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial007.front.sepia.ceph.com -o noatime 2026-02-22T19:51:17.846 DEBUG:teuthology.orchestra.run.trial007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2026-02-22T19:51:17.904 DEBUG:teuthology.orchestra.run.trial007:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2026-02-22T19:51:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:17.954 INFO:teuthology.orchestra.run.trial007.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:17.954 DEBUG:teuthology.orchestra.run.trial007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:18.016 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:18.013+0000 7fb7effb78c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2026-02-22T19:51:18.017 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:18.013+0000 7fb7effb78c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2026-02-22T19:51:18.017 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:18.013+0000 7fb7effb78c0 -1 bdev(0x564b792d9800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2026-02-22T19:51:18.017 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:18.013+0000 7fb7effb78c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2026-02-22T19:51:21.180 DEBUG:teuthology.orchestra.run.trial007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2026-02-22T19:51:21.232 DEBUG:teuthology.orchestra.run.trial007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:21.296 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:21.293+0000 7fe388d378c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2026-02-22T19:51:21.297 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:21.293+0000 7fe388d378c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2026-02-22T19:51:21.297 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:21.293+0000 7fe388d378c0 -1 bdev(0x563333817800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2026-02-22T19:51:21.297 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:21.293+0000 7fe388d378c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2026-02-22T19:51:24.380 DEBUG:teuthology.orchestra.run.trial007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2026-02-22T19:51:24.428 DEBUG:teuthology.orchestra.run.trial007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:24.492 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:24.489+0000 7fb090cc88c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2026-02-22T19:51:24.493 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:24.489+0000 7fb090cc88c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2026-02-22T19:51:24.493 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:24.489+0000 7fb090cc88c0 -1 bdev(0x5614e5b5d800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2026-02-22T19:51:24.493 INFO:teuthology.orchestra.run.trial007.stderr:2026-02-22T19:51:24.489+0000 7fb090cc88c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2026-02-22T19:51:27.656 DEBUG:teuthology.orchestra.run.trial007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2026-02-22T19:51:27.709 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-22T19:51:27.709 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:27.709 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2026-02-22T19:51:27.762 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:27.762 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2026-02-22T19:51:27.813 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:27.813 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2026-02-22T19:51:27.864 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:27.864 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2026-02-22T19:51:27.917 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:27.917 DEBUG:teuthology.orchestra.run.trial007:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2026-02-22T19:51:27.964 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:27.964 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2026-02-22T19:51:27.970 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:27.970 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2026-02-22T19:51:28.014 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:28.014 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2026-02-22T19:51:28.062 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:28.063 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2026-02-22T19:51:28.111 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:28.111 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2026-02-22T19:51:28.159 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:28.159 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2026-02-22T19:51:28.206 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:28.206 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2026-02-22T19:51:28.250 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:28.250 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2026-02-22T19:51:28.294 INFO:tasks.ceph:Adding keys to all mons... 2026-02-22T19:51:28.294 DEBUG:teuthology.orchestra.run.trial007:> sudo tee -a /etc/ceph/cluster1.keyring 2026-02-22T19:51:28.303 INFO:teuthology.orchestra.run.trial007.stdout:[mgr.x] 2026-02-22T19:51:28.303 INFO:teuthology.orchestra.run.trial007.stdout: key = AQAzXptpymPjARAAm0ZIYdJBbh0OLpDC0mq+dg== 2026-02-22T19:51:28.303 INFO:teuthology.orchestra.run.trial007.stdout:[osd.0] 2026-02-22T19:51:28.303 INFO:teuthology.orchestra.run.trial007.stdout: key = AQA2Xptp5ErlABAAWB4t4ypfaUgbSJccuYJLIg== 2026-02-22T19:51:28.303 INFO:teuthology.orchestra.run.trial007.stdout:[osd.1] 2026-02-22T19:51:28.303 INFO:teuthology.orchestra.run.trial007.stdout: key = AQA5XptplTeQERAAPYJjHOFnqnyhV3+LEpTz1w== 2026-02-22T19:51:28.303 INFO:teuthology.orchestra.run.trial007.stdout:[osd.2] 2026-02-22T19:51:28.304 INFO:teuthology.orchestra.run.trial007.stdout: key = AQA8XptpOSZCHRAABA6ziWE+0QIqQe8BCCh6dg== 2026-02-22T19:51:28.304 INFO:teuthology.orchestra.run.trial007.stdout:[client.0] 2026-02-22T19:51:28.304 INFO:teuthology.orchestra.run.trial007.stdout: key = AQAzXptpKIVoBRAAfHL1rdrvhBul9EMb+33IIw== 2026-02-22T19:51:28.304 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror] 2026-02-22T19:51:28.304 INFO:teuthology.orchestra.run.trial007.stdout: key = AQAzXptp2kIlBxAAZ+Y9S5ZbObyO8e1G5Zopng== 2026-02-22T19:51:28.304 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.0] 2026-02-22T19:51:28.304 INFO:teuthology.orchestra.run.trial007.stdout: key = AQAzXptp2MmRCxAAaTdk7uLqM5GHdCYSPqygXw== 2026-02-22T19:51:28.304 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.1] 2026-02-22T19:51:28.304 INFO:teuthology.orchestra.run.trial007.stdout: key = AQAzXptpHY3VDxAAOovzBkM8iTL+qviosOyUKw== 2026-02-22T19:51:28.304 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.2] 2026-02-22T19:51:28.304 INFO:teuthology.orchestra.run.trial007.stdout: key = AQAzXptpTuIaFBAAWgJJ82X1g9CQ3fkioHFLyA== 2026-02-22T19:51:28.305 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.3] 2026-02-22T19:51:28.305 INFO:teuthology.orchestra.run.trial007.stdout: key = AQAzXptpQqukGBAALSufzKqx2N65ZkN1j7QxMw== 2026-02-22T19:51:28.305 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.4] 2026-02-22T19:51:28.305 INFO:teuthology.orchestra.run.trial007.stdout: key = AQAzXptp0ojrHBAAHx2/fY2te7+QJNjJahM+1A== 2026-02-22T19:51:28.305 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.5] 2026-02-22T19:51:28.305 INFO:teuthology.orchestra.run.trial007.stdout: key = AQAzXptpi6E4IRAAPjW3HNUm1v7vuX2fd31Gdw== 2026-02-22T19:51:28.305 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.6] 2026-02-22T19:51:28.305 INFO:teuthology.orchestra.run.trial007.stdout: key = AQAzXptpIKSCJRAAYCbHRbh9rZvbFknjkYyaVA== 2026-02-22T19:51:28.306 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T19:51:28.369 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:51:28.433 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:51:28.496 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:51:28.560 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:28.625 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:28.689 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:28.752 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:28.817 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:28.881 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:28.944 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:29.009 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:29.073 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:51:29.137 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-22T19:51:29.137 DEBUG:teuthology.orchestra.run.trial007:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2026-02-22T19:51:29.188 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2026-02-22T19:51:29.257 DEBUG:teuthology.orchestra.run.trial007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2026-02-22T19:51:29.308 DEBUG:teuthology.orchestra.run.trial007:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2026-02-22T19:51:29.355 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2026-02-22T19:51:29.355 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2026-02-22T19:51:29.356 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2026-02-22T19:51:29.397 INFO:tasks.ceph.cluster1.mon.a:Started 2026-02-22T19:51:29.397 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2026-02-22T19:51:29.397 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2026-02-22T19:51:29.398 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2026-02-22T19:51:29.400 INFO:tasks.ceph.cluster1.mgr.x:Started 2026-02-22T19:51:29.400 INFO:tasks.ceph:Setting crush tunables to default 2026-02-22T19:51:29.400 DEBUG:teuthology.orchestra.run.trial007:> sudo ceph --cluster cluster1 osd crush tunables default 2026-02-22T19:51:29.534 INFO:teuthology.orchestra.run.trial007.stderr:adjusted tunables profile to default 2026-02-22T19:51:29.544 INFO:tasks.ceph:check_enable_crimson: False 2026-02-22T19:51:29.544 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2026-02-22T19:51:29.544 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:29.544 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2026-02-22T19:51:29.555 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:29.555 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2026-02-22T19:51:29.605 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:29.605 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2026-02-22T19:51:29.657 DEBUG:teuthology.orchestra.run.trial007:> sudo ceph --cluster cluster1 osd new 07d40a12-e882-4b68-af30-165696dfeff5 0 2026-02-22T19:51:29.724 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:29.721+0000 7f3086c47280 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-22T19:51:29.750 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:29.745+0000 7f3086c47280 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-22T19:51:29.787 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:29.781+0000 7f3086c47280 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-22T19:51:29.796 INFO:teuthology.orchestra.run.trial007.stdout:0 2026-02-22T19:51:29.805 DEBUG:teuthology.orchestra.run.trial007:> sudo ceph --cluster cluster1 osd new 3656b485-906a-4e26-908d-986c907b2604 1 2026-02-22T19:51:29.867 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:29.861+0000 7f3086c47280 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-22T19:51:29.947 INFO:teuthology.orchestra.run.trial007.stdout:1 2026-02-22T19:51:29.947 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:29.941+0000 7f3086c47280 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-22T19:51:29.957 DEBUG:teuthology.orchestra.run.trial007:> sudo ceph --cluster cluster1 osd new 2c972c13-b786-41d4-9f1b-ae529fc17749 2 2026-02-22T19:51:29.985 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:29.981+0000 7f3086c47280 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-22T19:51:30.061 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:30.057+0000 7f3086c47280 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-22T19:51:30.079 INFO:teuthology.orchestra.run.trial007.stdout:2 2026-02-22T19:51:30.089 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2026-02-22T19:51:30.089 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2026-02-22T19:51:30.091 INFO:tasks.ceph.cluster1.osd.0:Started 2026-02-22T19:51:30.091 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2026-02-22T19:51:30.091 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2026-02-22T19:51:30.093 INFO:tasks.ceph.cluster1.osd.1:Started 2026-02-22T19:51:30.093 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2026-02-22T19:51:30.094 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2026-02-22T19:51:30.096 INFO:tasks.ceph.cluster1.osd.2:Started 2026-02-22T19:51:30.096 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:30.194 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:30.194 INFO:teuthology.orchestra.run.trial007.stdout:{"epoch":5,"fsid":"a0d80545-8670-4b47-b26d-3047f07e2d2b","created":"2026-02-22T19:51:29.491054+0000","modified":"2026-02-22T19:51:30.076793+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:51:30.076793+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"07d40a12-e882-4b68-af30-165696dfeff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3656b485-906a-4e26-908d-986c907b2604","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2c972c13-b786-41d4-9f1b-ae529fc17749","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:30.204 INFO:tasks.ceph.ceph_manager.cluster1:[] 2026-02-22T19:51:30.205 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-22T19:51:30.240 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:30.237+0000 7f3086c47280 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-22T19:51:30.507 DEBUG:teuthology.orchestra.run.trial007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:30.544 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:30.541+0000 7f3086c47280 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-22T19:51:30.595 INFO:teuthology.misc.health.trial007.stdout: 2026-02-22T19:51:30.595 INFO:teuthology.misc.health.trial007.stdout:{"epoch":5,"fsid":"a0d80545-8670-4b47-b26d-3047f07e2d2b","created":"2026-02-22T19:51:29.491054+0000","modified":"2026-02-22T19:51:30.076793+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:51:30.076793+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"07d40a12-e882-4b68-af30-165696dfeff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3656b485-906a-4e26-908d-986c907b2604","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2c972c13-b786-41d4-9f1b-ae529fc17749","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:30.603 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T19:51:30.638 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:30.633+0000 7f3086c47280 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-22T19:51:30.666 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:30.661+0000 7f3086c47280 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-22T19:51:30.698 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:30.693+0000 7f3086c47280 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-22T19:51:30.741 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:30.737+0000 7f3086c47280 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-22T19:51:30.841 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:30.837+0000 7f3086c47280 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-22T19:51:31.035 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-22T19:51:31.207 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:31.201+0000 7f3086c47280 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-22T19:51:31.239 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:31.233+0000 7f3086c47280 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-22T19:51:31.332 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:31.325+0000 7f3086c47280 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-22T19:51:31.356 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:31.353+0000 7f3086c47280 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-22T19:51:31.383 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:31.377+0000 7f3086c47280 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-22T19:51:31.410 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:31.405+0000 7f3086c47280 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-22T19:51:31.531 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-22T19:51:31.531 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-22T19:51:31.532 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr: from numpy import show_config as show_numpy_config 2026-02-22T19:51:31.538 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:31.533+0000 7f3086c47280 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-22T19:51:31.565 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:31.561+0000 7f3086c47280 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-22T19:51:31.603 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:31.597+0000 7f3086c47280 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-22T19:51:31.661 INFO:tasks.ceph.cluster1.osd.0.trial007.stderr:2026-02-22T19:51:31.657+0000 7f4d7b9768c0 -1 Falling back to public interface 2026-02-22T19:51:31.677 INFO:tasks.ceph.cluster1.osd.1.trial007.stderr:2026-02-22T19:51:31.673+0000 7f2d0f7048c0 -1 Falling back to public interface 2026-02-22T19:51:32.121 INFO:tasks.ceph.cluster1.osd.2.trial007.stderr:2026-02-22T19:51:32.117+0000 7f10121488c0 -1 Falling back to public interface 2026-02-22T19:51:32.627 INFO:tasks.ceph.cluster1.mgr.x.trial007.stderr:2026-02-22T19:51:32.621+0000 7f2fa5699640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-22T19:51:35.504 INFO:tasks.ceph.cluster1.osd.0.trial007.stderr:2026-02-22T19:51:35.497+0000 7f4d7b9768c0 -1 osd.0 0 log_to_monitors true 2026-02-22T19:51:35.534 INFO:tasks.ceph.cluster1.osd.1.trial007.stderr:2026-02-22T19:51:35.529+0000 7f2d0f7048c0 -1 osd.1 0 log_to_monitors true 2026-02-22T19:51:36.245 INFO:tasks.ceph.cluster1.osd.2.trial007.stderr:2026-02-22T19:51:36.241+0000 7f10121488c0 -1 osd.2 0 log_to_monitors true 2026-02-22T19:51:36.633 INFO:tasks.ceph.cluster1.osd.1.trial007.stderr:2026-02-22T19:51:36.629+0000 7f2d0b6ad640 -1 osd.1 0 waiting for initial osdmap 2026-02-22T19:51:36.634 INFO:tasks.ceph.cluster1.osd.0.trial007.stderr:2026-02-22T19:51:36.629+0000 7f4d7791f640 -1 osd.0 0 waiting for initial osdmap 2026-02-22T19:51:36.649 INFO:tasks.ceph.cluster1.osd.1.trial007.stderr:2026-02-22T19:51:36.645+0000 7f2d034e0640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:51:36.669 INFO:tasks.ceph.cluster1.osd.0.trial007.stderr:2026-02-22T19:51:36.665+0000 7f4d6f752640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:51:36.906 DEBUG:teuthology.orchestra.run.trial007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:37.045 INFO:teuthology.misc.health.trial007.stdout: 2026-02-22T19:51:37.046 INFO:teuthology.misc.health.trial007.stdout:{"epoch":7,"fsid":"a0d80545-8670-4b47-b26d-3047f07e2d2b","created":"2026-02-22T19:51:29.491054+0000","modified":"2026-02-22T19:51:36.626772+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:51:30.076793+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"07d40a12-e882-4b68-af30-165696dfeff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3656b485-906a-4e26-908d-986c907b2604","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2c972c13-b786-41d4-9f1b-ae529fc17749","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:37.057 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T19:51:37.637 INFO:tasks.ceph.cluster1.osd.2.trial007.stderr:2026-02-22T19:51:37.633+0000 7f100e0f1640 -1 osd.2 0 waiting for initial osdmap 2026-02-22T19:51:37.661 INFO:tasks.ceph.cluster1.osd.2.trial007.stderr:2026-02-22T19:51:37.657+0000 7f1005f24640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:51:43.360 DEBUG:teuthology.orchestra.run.trial007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:43.496 INFO:teuthology.misc.health.trial007.stdout: 2026-02-22T19:51:43.497 INFO:teuthology.misc.health.trial007.stdout:{"epoch":11,"fsid":"a0d80545-8670-4b47-b26d-3047f07e2d2b","created":"2026-02-22T19:51:29.491054+0000","modified":"2026-02-22T19:51:40.638515+0000","last_up_change":"2026-02-22T19:51:38.632782+0000","last_in_change":"2026-02-22T19:51:30.076793+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:51:37.655569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"07d40a12-e882-4b68-af30-165696dfeff5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6802","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6803","nonce":344374375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6804","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6805","nonce":344374375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6808","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6809","nonce":344374375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6806","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6807","nonce":344374375}]},"public_addr":"10.20.193.7:6803/344374375","cluster_addr":"10.20.193.7:6805/344374375","heartbeat_back_addr":"10.20.193.7:6809/344374375","heartbeat_front_addr":"10.20.193.7:6807/344374375","state":["exists","up"]},{"osd":1,"uuid":"3656b485-906a-4e26-908d-986c907b2604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6810","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6811","nonce":2581020783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6812","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6813","nonce":2581020783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6816","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6817","nonce":2581020783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6814","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6815","nonce":2581020783}]},"public_addr":"10.20.193.7:6811/2581020783","cluster_addr":"10.20.193.7:6813/2581020783","heartbeat_back_addr":"10.20.193.7:6817/2581020783","heartbeat_front_addr":"10.20.193.7:6815/2581020783","state":["exists","up"]},{"osd":2,"uuid":"2c972c13-b786-41d4-9f1b-ae529fc17749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6818","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6819","nonce":461788605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6820","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6821","nonce":461788605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6824","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6825","nonce":461788605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6822","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6823","nonce":461788605}]},"public_addr":"10.20.193.7:6819/461788605","cluster_addr":"10.20.193.7:6821/461788605","heartbeat_back_addr":"10.20.193.7:6825/461788605","heartbeat_front_addr":"10.20.193.7:6823/461788605","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:36.538925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:36.494710+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:37.228811+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:43.506 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-22T19:51:43.506 INFO:tasks.ceph:Creating RBD pool 2026-02-22T19:51:43.506 DEBUG:teuthology.orchestra.run.trial007:> sudo ceph --cluster cluster1 osd pool create rbd 8 2026-02-22T19:51:44.646 INFO:teuthology.orchestra.run.trial007.stderr:pool 'rbd' created 2026-02-22T19:51:44.659 DEBUG:teuthology.orchestra.run.trial007:> rbd --cluster cluster1 pool init rbd 2026-02-22T19:51:47.657 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2026-02-22T19:51:47.657 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2026-02-22T19:51:47.658 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2026-02-22T19:51:47.658 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2026-02-22T19:51:47.658 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-22T19:51:47.819 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:47.828 INFO:teuthology.orchestra.run.trial007.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6800","nonce":2308269780},{"type":"v1","addr":"10.20.193.7:6801","nonce":2308269780}]},"active_addr":"10.20.193.7:6801/2308269780","active_change":"2026-02-22T19:51:31.619100+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.7:0","nonce":42969349}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.7:0","nonce":1224349843}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.7:0","nonce":194240002}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.7:0","nonce":1909206181}]}]} 2026-02-22T19:51:47.834 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2026-02-22T19:51:47.835 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2026-02-22T19:51:47.835 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:47.980 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:47.980 INFO:teuthology.orchestra.run.trial007.stdout:{"epoch":15,"fsid":"a0d80545-8670-4b47-b26d-3047f07e2d2b","created":"2026-02-22T19:51:29.491054+0000","modified":"2026-02-22T19:51:47.645213+0000","last_up_change":"2026-02-22T19:51:38.632782+0000","last_in_change":"2026-02-22T19:51:30.076793+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:51:37.655569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:51:43.649708+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"07d40a12-e882-4b68-af30-165696dfeff5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6802","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6803","nonce":344374375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6804","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6805","nonce":344374375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6808","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6809","nonce":344374375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6806","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6807","nonce":344374375}]},"public_addr":"10.20.193.7:6803/344374375","cluster_addr":"10.20.193.7:6805/344374375","heartbeat_back_addr":"10.20.193.7:6809/344374375","heartbeat_front_addr":"10.20.193.7:6807/344374375","state":["exists","up"]},{"osd":1,"uuid":"3656b485-906a-4e26-908d-986c907b2604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6810","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6811","nonce":2581020783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6812","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6813","nonce":2581020783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6816","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6817","nonce":2581020783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6814","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6815","nonce":2581020783}]},"public_addr":"10.20.193.7:6811/2581020783","cluster_addr":"10.20.193.7:6813/2581020783","heartbeat_back_addr":"10.20.193.7:6817/2581020783","heartbeat_front_addr":"10.20.193.7:6815/2581020783","state":["exists","up"]},{"osd":2,"uuid":"2c972c13-b786-41d4-9f1b-ae529fc17749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6818","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6819","nonce":461788605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6820","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6821","nonce":461788605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6824","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6825","nonce":461788605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6822","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6823","nonce":461788605}]},"public_addr":"10.20.193.7:6819/461788605","cluster_addr":"10.20.193.7:6821/461788605","heartbeat_back_addr":"10.20.193.7:6825/461788605","heartbeat_front_addr":"10.20.193.7:6823/461788605","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:36.538925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:36.494710+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:37.228811+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:47.990 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2026-02-22T19:51:47.991 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-22T19:51:48.132 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:48.133 INFO:teuthology.orchestra.run.trial007.stdout:{"epoch":15,"fsid":"a0d80545-8670-4b47-b26d-3047f07e2d2b","created":"2026-02-22T19:51:29.491054+0000","modified":"2026-02-22T19:51:47.645213+0000","last_up_change":"2026-02-22T19:51:38.632782+0000","last_in_change":"2026-02-22T19:51:30.076793+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:51:37.655569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:51:43.649708+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"07d40a12-e882-4b68-af30-165696dfeff5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6802","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6803","nonce":344374375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6804","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6805","nonce":344374375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6808","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6809","nonce":344374375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6806","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6807","nonce":344374375}]},"public_addr":"10.20.193.7:6803/344374375","cluster_addr":"10.20.193.7:6805/344374375","heartbeat_back_addr":"10.20.193.7:6809/344374375","heartbeat_front_addr":"10.20.193.7:6807/344374375","state":["exists","up"]},{"osd":1,"uuid":"3656b485-906a-4e26-908d-986c907b2604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6810","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6811","nonce":2581020783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6812","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6813","nonce":2581020783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6816","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6817","nonce":2581020783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6814","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6815","nonce":2581020783}]},"public_addr":"10.20.193.7:6811/2581020783","cluster_addr":"10.20.193.7:6813/2581020783","heartbeat_back_addr":"10.20.193.7:6817/2581020783","heartbeat_front_addr":"10.20.193.7:6815/2581020783","state":["exists","up"]},{"osd":2,"uuid":"2c972c13-b786-41d4-9f1b-ae529fc17749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6818","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6819","nonce":461788605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6820","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6821","nonce":461788605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6824","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6825","nonce":461788605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6822","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6823","nonce":461788605}]},"public_addr":"10.20.193.7:6819/461788605","cluster_addr":"10.20.193.7:6821/461788605","heartbeat_back_addr":"10.20.193.7:6825/461788605","heartbeat_front_addr":"10.20.193.7:6823/461788605","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:36.538925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:36.494710+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:37.228811+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:51:48.144 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2026-02-22T19:51:48.145 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2026-02-22T19:51:48.145 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2026-02-22T19:51:48.229 INFO:teuthology.orchestra.run.trial007.stdout:38654705669 2026-02-22T19:51:48.230 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2026-02-22T19:51:48.231 INFO:teuthology.orchestra.run.trial007.stdout:34359738371 2026-02-22T19:51:48.231 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2026-02-22T19:51:48.232 INFO:teuthology.orchestra.run.trial007.stdout:34359738372 2026-02-22T19:51:48.232 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2026-02-22T19:51:48.414 INFO:teuthology.orchestra.run.trial007.stdout:34359738370 2026-02-22T19:51:48.415 INFO:teuthology.orchestra.run.trial007.stdout:38654705668 2026-02-22T19:51:48.419 INFO:teuthology.orchestra.run.trial007.stdout:34359738371 2026-02-22T19:51:48.426 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.0 2026-02-22T19:51:48.427 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705668 for osd.2 2026-02-22T19:51:48.432 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2026-02-22T19:51:49.427 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2026-02-22T19:51:49.428 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2026-02-22T19:51:49.433 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2026-02-22T19:51:49.573 INFO:teuthology.orchestra.run.trial007.stdout:38654705668 2026-02-22T19:51:49.576 INFO:teuthology.orchestra.run.trial007.stdout:34359738370 2026-02-22T19:51:49.586 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705668 for osd.2 2026-02-22T19:51:49.587 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.0 2026-02-22T19:51:49.628 INFO:teuthology.orchestra.run.trial007.stdout:34359738371 2026-02-22T19:51:49.638 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2026-02-22T19:51:50.586 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2026-02-22T19:51:50.588 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2026-02-22T19:51:50.639 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2026-02-22T19:51:50.732 INFO:teuthology.orchestra.run.trial007.stdout:34359738372 2026-02-22T19:51:50.734 INFO:teuthology.orchestra.run.trial007.stdout:38654705669 2026-02-22T19:51:50.745 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738372 for osd.0 2026-02-22T19:51:50.745 DEBUG:teuthology.parallel:result is None 2026-02-22T19:51:50.746 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705669 for osd.2 2026-02-22T19:51:50.746 DEBUG:teuthology.parallel:result is None 2026-02-22T19:51:50.783 INFO:teuthology.orchestra.run.trial007.stdout:34359738373 2026-02-22T19:51:50.794 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738373 for osd.1 2026-02-22T19:51:50.794 DEBUG:teuthology.parallel:result is None 2026-02-22T19:51:50.794 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2026-02-22T19:51:50.795 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T19:51:50.978 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:50.978 INFO:teuthology.orchestra.run.trial007.stderr:dumped all 2026-02-22T19:51:50.988 INFO:teuthology.orchestra.run.trial007.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2026-02-22T19:51:49.623327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81388,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034132,"statfs":{"total":289910292480,"available":289826951168,"internally_reserved":0,"allocated":499712,"data_stored":1289748,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.978703"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.650108+0000","last_change":"2026-02-22T19:51:47.650108+0000","last_active":"2026-02-22T19:51:47.650108+0000","last_peered":"2026-02-22T19:51:47.650108+0000","last_clean":"2026-02-22T19:51:47.650108+0000","last_became_active":"2026-02-22T19:51:45.650713+0000","last_became_peered":"2026-02-22T19:51:45.650713+0000","last_unstale":"2026-02-22T19:51:47.650108+0000","last_undegraded":"2026-02-22T19:51:47.650108+0000","last_fullsized":"2026-02-22T19:51:47.650108+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:19:42.205895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000847958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.649914+0000","last_change":"2026-02-22T19:51:47.649914+0000","last_active":"2026-02-22T19:51:47.649914+0000","last_peered":"2026-02-22T19:51:47.649914+0000","last_clean":"2026-02-22T19:51:47.649914+0000","last_became_active":"2026-02-22T19:51:45.650166+0000","last_became_peered":"2026-02-22T19:51:45.650166+0000","last_unstale":"2026-02-22T19:51:47.649914+0000","last_undegraded":"2026-02-22T19:51:47.649914+0000","last_fullsized":"2026-02-22T19:51:47.649914+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:36:23.935956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071726600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.649520+0000","last_change":"2026-02-22T19:51:47.649520+0000","last_active":"2026-02-22T19:51:47.649520+0000","last_peered":"2026-02-22T19:51:47.649520+0000","last_clean":"2026-02-22T19:51:47.649520+0000","last_became_active":"2026-02-22T19:51:45.650286+0000","last_became_peered":"2026-02-22T19:51:45.650286+0000","last_unstale":"2026-02-22T19:51:47.649520+0000","last_undegraded":"2026-02-22T19:51:47.649520+0000","last_fullsized":"2026-02-22T19:51:47.649520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:58:53.225941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039283599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.649526+0000","last_change":"2026-02-22T19:51:47.649526+0000","last_active":"2026-02-22T19:51:47.649526+0000","last_peered":"2026-02-22T19:51:47.649526+0000","last_clean":"2026-02-22T19:51:47.649526+0000","last_became_active":"2026-02-22T19:51:45.649840+0000","last_became_peered":"2026-02-22T19:51:45.649840+0000","last_unstale":"2026-02-22T19:51:47.649526+0000","last_undegraded":"2026-02-22T19:51:47.649526+0000","last_fullsized":"2026-02-22T19:51:47.649526+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:32:49.997988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042606799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.648420+0000","last_change":"2026-02-22T19:51:47.647395+0000","last_active":"2026-02-22T19:51:47.648420+0000","last_peered":"2026-02-22T19:51:47.648420+0000","last_clean":"2026-02-22T19:51:47.648420+0000","last_became_active":"2026-02-22T19:51:45.650504+0000","last_became_peered":"2026-02-22T19:51:45.650504+0000","last_unstale":"2026-02-22T19:51:47.648420+0000","last_undegraded":"2026-02-22T19:51:47.648420+0000","last_fullsized":"2026-02-22T19:51:47.648420+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:44:15.940372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033513899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.129796+0000","last_change":"2026-02-22T19:51:48.129796+0000","last_active":"2026-02-22T19:51:48.129796+0000","last_peered":"2026-02-22T19:51:48.129796+0000","last_clean":"2026-02-22T19:51:48.129796+0000","last_became_active":"2026-02-22T19:51:45.650606+0000","last_became_peered":"2026-02-22T19:51:45.650606+0000","last_unstale":"2026-02-22T19:51:48.129796+0000","last_undegraded":"2026-02-22T19:51:48.129796+0000","last_fullsized":"2026-02-22T19:51:48.129796+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:23:01.064130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000278965,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.129782+0000","last_change":"2026-02-22T19:51:48.129782+0000","last_active":"2026-02-22T19:51:48.129782+0000","last_peered":"2026-02-22T19:51:48.129782+0000","last_clean":"2026-02-22T19:51:48.129782+0000","last_became_active":"2026-02-22T19:51:45.650544+0000","last_became_peered":"2026-02-22T19:51:45.650544+0000","last_unstale":"2026-02-22T19:51:48.129782+0000","last_undegraded":"2026-02-22T19:51:48.129782+0000","last_fullsized":"2026-02-22T19:51:48.129782+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:48:50.952142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027495800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.649910+0000","last_change":"2026-02-22T19:51:47.649910+0000","last_active":"2026-02-22T19:51:47.649910+0000","last_peered":"2026-02-22T19:51:47.649910+0000","last_clean":"2026-02-22T19:51:47.649910+0000","last_became_active":"2026-02-22T19:51:45.649866+0000","last_became_peered":"2026-02-22T19:51:45.649866+0000","last_unstale":"2026-02-22T19:51:47.649910+0000","last_undegraded":"2026-02-22T19:51:47.649910+0000","last_fullsized":"2026-02-22T19:51:47.649910+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:51:55.975957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071026300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'72","reported_seq":110,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.648901+0000","last_change":"2026-02-22T19:51:40.106594+0000","last_active":"2026-02-22T19:51:47.648901+0000","last_peered":"2026-02-22T19:51:47.648901+0000","last_clean":"2026-02-22T19:51:47.648901+0000","last_became_active":"2026-02-22T19:51:40.105788+0000","last_became_peered":"2026-02-22T19:51:40.105788+0000","last_unstale":"2026-02-22T19:51:47.648901+0000","last_undegraded":"2026-02-22T19:51:47.648901+0000","last_fullsized":"2026-02-22T19:51:47.648901+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_clean_scrub_stamp":"2026-02-22T19:51:38.632782+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:04:29.975974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344764,"statfs":{"total":96636764160,"available":96609038336,"internally_reserved":0,"allocated":114688,"data_stored":31634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":192512,"data_stored":629057,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":192512,"data_stored":629057,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T19:51:50.989 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T19:51:51.138 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:51.139 INFO:teuthology.orchestra.run.trial007.stderr:dumped all 2026-02-22T19:51:51.148 INFO:teuthology.orchestra.run.trial007.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2026-02-22T19:51:49.623327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81388,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034132,"statfs":{"total":289910292480,"available":289826951168,"internally_reserved":0,"allocated":499712,"data_stored":1289748,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.978703"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.650108+0000","last_change":"2026-02-22T19:51:47.650108+0000","last_active":"2026-02-22T19:51:47.650108+0000","last_peered":"2026-02-22T19:51:47.650108+0000","last_clean":"2026-02-22T19:51:47.650108+0000","last_became_active":"2026-02-22T19:51:45.650713+0000","last_became_peered":"2026-02-22T19:51:45.650713+0000","last_unstale":"2026-02-22T19:51:47.650108+0000","last_undegraded":"2026-02-22T19:51:47.650108+0000","last_fullsized":"2026-02-22T19:51:47.650108+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:19:42.205895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000847958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.649914+0000","last_change":"2026-02-22T19:51:47.649914+0000","last_active":"2026-02-22T19:51:47.649914+0000","last_peered":"2026-02-22T19:51:47.649914+0000","last_clean":"2026-02-22T19:51:47.649914+0000","last_became_active":"2026-02-22T19:51:45.650166+0000","last_became_peered":"2026-02-22T19:51:45.650166+0000","last_unstale":"2026-02-22T19:51:47.649914+0000","last_undegraded":"2026-02-22T19:51:47.649914+0000","last_fullsized":"2026-02-22T19:51:47.649914+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:36:23.935956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071726600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.649520+0000","last_change":"2026-02-22T19:51:47.649520+0000","last_active":"2026-02-22T19:51:47.649520+0000","last_peered":"2026-02-22T19:51:47.649520+0000","last_clean":"2026-02-22T19:51:47.649520+0000","last_became_active":"2026-02-22T19:51:45.650286+0000","last_became_peered":"2026-02-22T19:51:45.650286+0000","last_unstale":"2026-02-22T19:51:47.649520+0000","last_undegraded":"2026-02-22T19:51:47.649520+0000","last_fullsized":"2026-02-22T19:51:47.649520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:58:53.225941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039283599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.649526+0000","last_change":"2026-02-22T19:51:47.649526+0000","last_active":"2026-02-22T19:51:47.649526+0000","last_peered":"2026-02-22T19:51:47.649526+0000","last_clean":"2026-02-22T19:51:47.649526+0000","last_became_active":"2026-02-22T19:51:45.649840+0000","last_became_peered":"2026-02-22T19:51:45.649840+0000","last_unstale":"2026-02-22T19:51:47.649526+0000","last_undegraded":"2026-02-22T19:51:47.649526+0000","last_fullsized":"2026-02-22T19:51:47.649526+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:32:49.997988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042606799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.648420+0000","last_change":"2026-02-22T19:51:47.647395+0000","last_active":"2026-02-22T19:51:47.648420+0000","last_peered":"2026-02-22T19:51:47.648420+0000","last_clean":"2026-02-22T19:51:47.648420+0000","last_became_active":"2026-02-22T19:51:45.650504+0000","last_became_peered":"2026-02-22T19:51:45.650504+0000","last_unstale":"2026-02-22T19:51:47.648420+0000","last_undegraded":"2026-02-22T19:51:47.648420+0000","last_fullsized":"2026-02-22T19:51:47.648420+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:44:15.940372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033513899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.129796+0000","last_change":"2026-02-22T19:51:48.129796+0000","last_active":"2026-02-22T19:51:48.129796+0000","last_peered":"2026-02-22T19:51:48.129796+0000","last_clean":"2026-02-22T19:51:48.129796+0000","last_became_active":"2026-02-22T19:51:45.650606+0000","last_became_peered":"2026-02-22T19:51:45.650606+0000","last_unstale":"2026-02-22T19:51:48.129796+0000","last_undegraded":"2026-02-22T19:51:48.129796+0000","last_fullsized":"2026-02-22T19:51:48.129796+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:23:01.064130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000278965,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:48.129782+0000","last_change":"2026-02-22T19:51:48.129782+0000","last_active":"2026-02-22T19:51:48.129782+0000","last_peered":"2026-02-22T19:51:48.129782+0000","last_clean":"2026-02-22T19:51:48.129782+0000","last_became_active":"2026-02-22T19:51:45.650544+0000","last_became_peered":"2026-02-22T19:51:45.650544+0000","last_unstale":"2026-02-22T19:51:48.129782+0000","last_undegraded":"2026-02-22T19:51:48.129782+0000","last_fullsized":"2026-02-22T19:51:48.129782+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:48:50.952142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027495800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.649910+0000","last_change":"2026-02-22T19:51:47.649910+0000","last_active":"2026-02-22T19:51:47.649910+0000","last_peered":"2026-02-22T19:51:47.649910+0000","last_clean":"2026-02-22T19:51:47.649910+0000","last_became_active":"2026-02-22T19:51:45.649866+0000","last_became_peered":"2026-02-22T19:51:45.649866+0000","last_unstale":"2026-02-22T19:51:47.649910+0000","last_undegraded":"2026-02-22T19:51:47.649910+0000","last_fullsized":"2026-02-22T19:51:47.649910+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:51:55.975957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071026300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'72","reported_seq":110,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:51:47.648901+0000","last_change":"2026-02-22T19:51:40.106594+0000","last_active":"2026-02-22T19:51:47.648901+0000","last_peered":"2026-02-22T19:51:47.648901+0000","last_clean":"2026-02-22T19:51:47.648901+0000","last_became_active":"2026-02-22T19:51:40.105788+0000","last_became_peered":"2026-02-22T19:51:40.105788+0000","last_unstale":"2026-02-22T19:51:47.648901+0000","last_undegraded":"2026-02-22T19:51:47.648901+0000","last_fullsized":"2026-02-22T19:51:47.648901+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_clean_scrub_stamp":"2026-02-22T19:51:38.632782+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:04:29.975974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344764,"statfs":{"total":96636764160,"available":96609038336,"internally_reserved":0,"allocated":114688,"data_stored":31634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":192512,"data_stored":629057,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":192512,"data_stored":629057,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T19:51:51.149 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2026-02-22T19:51:51.149 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2026-02-22T19:51:51.150 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2026-02-22T19:51:51.150 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2026-02-22T19:51:51.315 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:51.316 INFO:teuthology.orchestra.run.trial007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-22T19:51:51.326 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2026-02-22T19:51:51.326 INFO:teuthology.run_tasks:Running task ceph... 2026-02-22T19:51:51.340 INFO:tasks.ceph:Creating ceph cluster cluster2... 2026-02-22T19:51:51.340 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-22T19:51:51.340 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974', 'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-journaling}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '63974', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror.sh workunit to test the rbd-mirror daemon in snapshot mode'}], 'name': 'yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9196, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'targets': {'trial007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJMUlU4Cteug0GiCx5j2Uet2DqUnnKtQrIB7U1GQif+JQTZksVOpHfZDaz1UwK4wK9qYTa3nRdx4FZj5aIdfkyo=', 'trial017.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLDW5vyj+GviZNpv6neyV3hYsSiL3/fKOpVwfikmufzMXNVE4sj5SIJFR4lUNFi/6pYMJ2OdRg0h3yjhc8eG8CM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,journaling', 'RBD_MIRROR_CONFIG_KEY': '1', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-22_19:35:38', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-22T19:51:51.341 DEBUG:teuthology.orchestra.run.trial007:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2026-02-22T19:51:51.343 DEBUG:teuthology.orchestra.run.trial017:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2026-02-22T19:51:51.349 DEBUG:teuthology.orchestra.run.trial007:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T19:51:51.393 DEBUG:teuthology.orchestra.run.trial017:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T19:51:51.402 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:51.403 DEBUG:teuthology.orchestra.run.trial017:> dd if=/scratch_devs of=/dev/stdout 2026-02-22T19:51:51.447 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-22T19:51:51.447 DEBUG:teuthology.orchestra.run.trial017:> stat /dev/vg_nvme/lv_1 2026-02-22T19:51:51.494 INFO:teuthology.orchestra.run.trial017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-22T19:51:51.494 INFO:teuthology.orchestra.run.trial017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:51.494 INFO:teuthology.orchestra.run.trial017.stdout:Device: 5h/5d Inode: 898 Links: 1 2026-02-22T19:51:51.494 INFO:teuthology.orchestra.run.trial017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:51.495 INFO:teuthology.orchestra.run.trial017.stdout:Access: 2026-02-22 19:46:39.856950272 +0000 2026-02-22T19:51:51.495 INFO:teuthology.orchestra.run.trial017.stdout:Modify: 2026-02-22 19:46:39.704952856 +0000 2026-02-22T19:51:51.495 INFO:teuthology.orchestra.run.trial017.stdout:Change: 2026-02-22 19:46:39.704952856 +0000 2026-02-22T19:51:51.495 INFO:teuthology.orchestra.run.trial017.stdout: Birth: - 2026-02-22T19:51:51.495 DEBUG:teuthology.orchestra.run.trial017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-22T19:51:51.541 INFO:teuthology.orchestra.run.trial017.stderr:1+0 records in 2026-02-22T19:51:51.541 INFO:teuthology.orchestra.run.trial017.stderr:1+0 records out 2026-02-22T19:51:51.542 INFO:teuthology.orchestra.run.trial017.stderr:512 bytes copied, 0.000176808 s, 2.9 MB/s 2026-02-22T19:51:51.543 DEBUG:teuthology.orchestra.run.trial017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-22T19:51:51.592 DEBUG:teuthology.orchestra.run.trial017:> stat /dev/vg_nvme/lv_2 2026-02-22T19:51:51.638 INFO:teuthology.orchestra.run.trial017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-22T19:51:51.638 INFO:teuthology.orchestra.run.trial017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:51.638 INFO:teuthology.orchestra.run.trial017.stdout:Device: 5h/5d Inode: 922 Links: 1 2026-02-22T19:51:51.639 INFO:teuthology.orchestra.run.trial017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:51.639 INFO:teuthology.orchestra.run.trial017.stdout:Access: 2026-02-22 19:46:40.184944698 +0000 2026-02-22T19:51:51.639 INFO:teuthology.orchestra.run.trial017.stdout:Modify: 2026-02-22 19:46:40.020947485 +0000 2026-02-22T19:51:51.639 INFO:teuthology.orchestra.run.trial017.stdout:Change: 2026-02-22 19:46:40.020947485 +0000 2026-02-22T19:51:51.639 INFO:teuthology.orchestra.run.trial017.stdout: Birth: - 2026-02-22T19:51:51.639 DEBUG:teuthology.orchestra.run.trial017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-22T19:51:51.685 INFO:teuthology.orchestra.run.trial017.stderr:1+0 records in 2026-02-22T19:51:51.686 INFO:teuthology.orchestra.run.trial017.stderr:1+0 records out 2026-02-22T19:51:51.686 INFO:teuthology.orchestra.run.trial017.stderr:512 bytes copied, 0.000174343 s, 2.9 MB/s 2026-02-22T19:51:51.687 DEBUG:teuthology.orchestra.run.trial017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-22T19:51:51.736 DEBUG:teuthology.orchestra.run.trial017:> stat /dev/vg_nvme/lv_3 2026-02-22T19:51:51.782 INFO:teuthology.orchestra.run.trial017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-22T19:51:51.782 INFO:teuthology.orchestra.run.trial017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:51.783 INFO:teuthology.orchestra.run.trial017.stdout:Device: 5h/5d Inode: 945 Links: 1 2026-02-22T19:51:51.783 INFO:teuthology.orchestra.run.trial017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:51.783 INFO:teuthology.orchestra.run.trial017.stdout:Access: 2026-02-22 19:46:40.340942047 +0000 2026-02-22T19:51:51.783 INFO:teuthology.orchestra.run.trial017.stdout:Modify: 2026-02-22 19:46:40.336942113 +0000 2026-02-22T19:51:51.783 INFO:teuthology.orchestra.run.trial017.stdout:Change: 2026-02-22 19:46:40.336942113 +0000 2026-02-22T19:51:51.783 INFO:teuthology.orchestra.run.trial017.stdout: Birth: - 2026-02-22T19:51:51.783 DEBUG:teuthology.orchestra.run.trial017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-22T19:51:51.829 INFO:teuthology.orchestra.run.trial017.stderr:1+0 records in 2026-02-22T19:51:51.830 INFO:teuthology.orchestra.run.trial017.stderr:1+0 records out 2026-02-22T19:51:51.830 INFO:teuthology.orchestra.run.trial017.stderr:512 bytes copied, 0.000170245 s, 3.0 MB/s 2026-02-22T19:51:51.831 DEBUG:teuthology.orchestra.run.trial017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-22T19:51:51.880 DEBUG:teuthology.orchestra.run.trial017:> stat /dev/vg_nvme/lv_4 2026-02-22T19:51:51.926 INFO:teuthology.orchestra.run.trial017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-22T19:51:51.927 INFO:teuthology.orchestra.run.trial017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T19:51:51.927 INFO:teuthology.orchestra.run.trial017.stdout:Device: 5h/5d Inode: 969 Links: 1 2026-02-22T19:51:51.927 INFO:teuthology.orchestra.run.trial017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T19:51:51.927 INFO:teuthology.orchestra.run.trial017.stdout:Access: 2026-02-22 19:46:40.824933819 +0000 2026-02-22T19:51:51.927 INFO:teuthology.orchestra.run.trial017.stdout:Modify: 2026-02-22 19:46:40.656936676 +0000 2026-02-22T19:51:51.927 INFO:teuthology.orchestra.run.trial017.stdout:Change: 2026-02-22 19:46:40.656936676 +0000 2026-02-22T19:51:51.927 INFO:teuthology.orchestra.run.trial017.stdout: Birth: - 2026-02-22T19:51:51.927 DEBUG:teuthology.orchestra.run.trial017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-22T19:51:51.977 INFO:teuthology.orchestra.run.trial017.stderr:1+0 records in 2026-02-22T19:51:51.977 INFO:teuthology.orchestra.run.trial017.stderr:1+0 records out 2026-02-22T19:51:51.977 INFO:teuthology.orchestra.run.trial017.stderr:512 bytes copied, 0.000179412 s, 2.9 MB/s 2026-02-22T19:51:51.978 DEBUG:teuthology.orchestra.run.trial017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-22T19:51:52.028 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:52.028 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial017.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T19:51:52.028 INFO:tasks.ceph:Generating config... 2026-02-22T19:51:52.030 INFO:tasks.ceph:[client] debug rbd = 20 2026-02-22T19:51:52.030 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2026-02-22T19:51:52.030 INFO:tasks.ceph:[client] log to stderr = False 2026-02-22T19:51:52.030 INFO:tasks.ceph:[client] rbd default features = 125 2026-02-22T19:51:52.030 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.031 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.031 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.031 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.031 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.031 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.031 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.031 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.031 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.031 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.031 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.031 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.032 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.032 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.032 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-22T19:51:52.032 INFO:tasks.ceph:[global] mon warn on pool no app = False 2026-02-22T19:51:52.032 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2026-02-22T19:51:52.032 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-22T19:51:52.032 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-22T19:51:52.032 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-22T19:51:52.032 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-22T19:51:52.032 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-22T19:51:52.032 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-22T19:51:52.033 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-22T19:51:52.034 INFO:tasks.ceph:Setting up cluster2.mon.a... 2026-02-22T19:51:52.034 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2026-02-22T19:51:52.086 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster2.keyring 2026-02-22T19:51:52.089 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2026-02-22T19:51:52.149 DEBUG:teuthology.orchestra.run.trial017:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-22T19:51:52.198 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '10.20.193.17')] 2026-02-22T19:51:52.199 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.17', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2026-02-22T19:51:52.200 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:52.200 DEBUG:teuthology.orchestra.run.trial017:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T19:51:52.243 DEBUG:teuthology.orchestra.run.trial017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.17 --print /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:52.303 INFO:teuthology.orchestra.run.trial017.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:52.304 INFO:teuthology.orchestra.run.trial017.stdout:monmaptool: generated fsid 10943a12-2d65-41e2-884e-f5f3b2ca910b 2026-02-22T19:51:52.304 INFO:teuthology.orchestra.run.trial017.stdout:setting min_mon_release = quincy 2026-02-22T19:51:52.304 INFO:teuthology.orchestra.run.trial017.stdout:epoch 0 2026-02-22T19:51:52.304 INFO:teuthology.orchestra.run.trial017.stdout:fsid 10943a12-2d65-41e2-884e-f5f3b2ca910b 2026-02-22T19:51:52.304 INFO:teuthology.orchestra.run.trial017.stdout:last_changed 2026-02-22T19:51:52.304290+0000 2026-02-22T19:51:52.304 INFO:teuthology.orchestra.run.trial017.stdout:created 2026-02-22T19:51:52.304290+0000 2026-02-22T19:51:52.304 INFO:teuthology.orchestra.run.trial017.stdout:min_mon_release 17 (quincy) 2026-02-22T19:51:52.304 INFO:teuthology.orchestra.run.trial017.stdout:election_strategy: 1 2026-02-22T19:51:52.304 INFO:teuthology.orchestra.run.trial017.stdout:0: [v2:10.20.193.17:3300/0,v1:10.20.193.17:6789/0] mon.a 2026-02-22T19:51:52.304 INFO:teuthology.orchestra.run.trial017.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2026-02-22T19:51:52.306 DEBUG:teuthology.orchestra.run.trial017:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T19:51:52.350 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 10943a12-2d65-41e2-884e-f5f3b2ca910b... 2026-02-22T19:51:52.352 DEBUG:teuthology.orchestra.run.trial007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2026-02-22T19:51:52.354 DEBUG:teuthology.orchestra.run.trial017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2026-02-22T19:51:52.393 INFO:teuthology.orchestra.run.trial007.stdout:[global] 2026-02-22T19:51:52.393 INFO:teuthology.orchestra.run.trial007.stdout: chdir = "" 2026-02-22T19:51:52.393 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T19:51:52.393 INFO:teuthology.orchestra.run.trial007.stdout: auth supported = cephx 2026-02-22T19:51:52.393 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.393 INFO:teuthology.orchestra.run.trial007.stdout: filestore xattr use omap = true 2026-02-22T19:51:52.393 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.393 INFO:teuthology.orchestra.run.trial007.stdout: mon clock drift allowed = 1.000 2026-02-22T19:51:52.393 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.394 INFO:teuthology.orchestra.run.trial007.stdout: osd crush chooseleaf type = 0 2026-02-22T19:51:52.394 INFO:teuthology.orchestra.run.trial007.stdout: auth debug = true 2026-02-22T19:51:52.394 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.394 INFO:teuthology.orchestra.run.trial007.stdout: ms die on old message = true 2026-02-22T19:51:52.394 INFO:teuthology.orchestra.run.trial007.stdout: ms die on bug = true 2026-02-22T19:51:52.394 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.394 INFO:teuthology.orchestra.run.trial007.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T19:51:52.394 INFO:teuthology.orchestra.run.trial007.stdout: mon pg warn max object skew = 0 2026-02-22T19:51:52.394 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.394 INFO:teuthology.orchestra.run.trial007.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T19:51:52.394 INFO:teuthology.orchestra.run.trial007.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: osd pool default size = 2 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: mon osd allow primary affinity = true 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: mon osd allow pg remap = true 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on legacy crush tunables = false 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on crush straw calc version zero = false 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on no sortbitwise = false 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on osd down out interval zero = false 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on too few osds = false 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T19:51:52.395 INFO:teuthology.orchestra.run.trial007.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T19:51:52.396 INFO:teuthology.orchestra.run.trial007.stdout: mon_allow_pool_size_one = true 2026-02-22T19:51:52.396 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.396 INFO:teuthology.orchestra.run.trial007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T19:51:52.396 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.396 INFO:teuthology.orchestra.run.trial007.stdout: osd default data pool replay window = 5 2026-02-22T19:51:52.396 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.396 INFO:teuthology.orchestra.run.trial007.stdout: mon allow pool delete = true 2026-02-22T19:51:52.396 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.396 INFO:teuthology.orchestra.run.trial007.stdout: mon cluster log file level = debug 2026-02-22T19:51:52.396 INFO:teuthology.orchestra.run.trial007.stdout: debug asserts on shutdown = true 2026-02-22T19:51:52.396 INFO:teuthology.orchestra.run.trial007.stdout: mon health detail to clog = false 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout: mon host = 10.20.193.17 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout: mon client directed command retry = 5 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout: mon warn on pool no app = False 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout: ms inject socket failures = 5000 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout: fsid = 10943a12-2d65-41e2-884e-f5f3b2ca910b 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout:[osd] 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout: osd journal size = 100 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout: osd scrub load threshold = 5.0 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout: osd scrub max interval = 600 2026-02-22T19:51:52.397 INFO:teuthology.orchestra.run.trial007.stdout: osd mclock profile = high_recovery_ops 2026-02-22T19:51:52.398 INFO:teuthology.orchestra.run.trial007.stdout: osd mclock skip benchmark = true 2026-02-22T19:51:52.398 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.398 INFO:teuthology.orchestra.run.trial007.stdout: osd recover clone overlap = true 2026-02-22T19:51:52.398 INFO:teuthology.orchestra.run.trial007.stdout: osd recovery max chunk = 1048576 2026-02-22T19:51:52.398 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.398 INFO:teuthology.orchestra.run.trial007.stdout: osd debug shutdown = true 2026-02-22T19:51:52.398 INFO:teuthology.orchestra.run.trial007.stdout: osd debug op order = true 2026-02-22T19:51:52.398 INFO:teuthology.orchestra.run.trial007.stdout: osd debug verify stray on activate = true 2026-02-22T19:51:52.398 INFO:teuthology.orchestra.run.trial007.stdout: osd debug trim objects = true 2026-02-22T19:51:52.398 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.398 INFO:teuthology.orchestra.run.trial007.stdout: osd open classes on start = true 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: osd debug pg log writeout = true 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: osd deep scrub update digest min age = 30 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: osd map max advance = 10 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: journal zero on create = true 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: filestore ondisk finisher threads = 3 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: filestore apply finisher threads = 3 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.399 INFO:teuthology.orchestra.run.trial007.stdout: bdev debug aio = true 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: osd debug misdirected ops = true 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: bluestore block size = 96636764160 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: bluestore compression algorithm = zlib 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: bluestore compression mode = aggressive 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: bluestore fsck on mount = True 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: debug bluefs = 1/20 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: debug bluestore = 1/20 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: debug ms = 1 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: debug osd = 20 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: debug rocksdb = 4/10 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:52.400 INFO:teuthology.orchestra.run.trial007.stdout: mon osd full ratio = 0.9 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: mon osd nearfull ratio = 0.8 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: osd failsafe full ratio = 0.95 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: osd objectstore = bluestore 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout:[mgr] 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: debug ms = 1 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: debug mgr = 20 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: debug mon = 20 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: debug auth = 20 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: mgr/telemetry/nag = false 2026-02-22T19:51:52.401 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.402 INFO:teuthology.orchestra.run.trial007.stdout:[mon] 2026-02-22T19:51:52.402 INFO:teuthology.orchestra.run.trial007.stdout: debug ms = 1 2026-02-22T19:51:52.402 INFO:teuthology.orchestra.run.trial007.stdout: debug mon = 20 2026-02-22T19:51:52.402 INFO:teuthology.orchestra.run.trial007.stdout: debug paxos = 20 2026-02-22T19:51:52.402 INFO:teuthology.orchestra.run.trial007.stdout: debug auth = 20 2026-02-22T19:51:52.402 INFO:teuthology.orchestra.run.trial007.stdout: mon data avail warn = 5 2026-02-22T19:51:52.402 INFO:teuthology.orchestra.run.trial007.stdout: mon mgr mkfs grace = 240 2026-02-22T19:51:52.402 INFO:teuthology.orchestra.run.trial007.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:52.402 INFO:teuthology.orchestra.run.trial007.stdout: mon osd reporter subtree level = osd 2026-02-22T19:51:52.402 INFO:teuthology.orchestra.run.trial007.stdout: mon osd prime pg temp = true 2026-02-22T19:51:52.402 INFO:teuthology.orchestra.run.trial007.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: # 1m isn't quite enough 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: mon_down_mkfs_grace = 2m 2026-02-22T19:51:52.403 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout: mon_warn_on_filestore_osds = false 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout:[client] 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout: rgw cache enabled = true 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout: rgw enable ops log = true 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout: rgw enable usage log = true 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout: debug rbd = 20 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout: debug rbd_mirror = 15 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout: log to stderr = False 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout: rbd default features = 125 2026-02-22T19:51:52.404 INFO:teuthology.orchestra.run.trial007.stdout:[mon.a] 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.0] 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.1] 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.2] 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.3] 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.405 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.4] 2026-02-22T19:51:52.406 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.406 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.406 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.5] 2026-02-22T19:51:52.406 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.406 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.406 INFO:teuthology.orchestra.run.trial007.stdout:[client.mirror.6] 2026-02-22T19:51:52.406 INFO:teuthology.orchestra.run.trial007.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.406 INFO:teuthology.orchestra.run.trial007.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.408 INFO:teuthology.orchestra.run.trial017.stdout:[global] 2026-02-22T19:51:52.408 INFO:teuthology.orchestra.run.trial017.stdout: chdir = "" 2026-02-22T19:51:52.408 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T19:51:52.408 INFO:teuthology.orchestra.run.trial017.stdout: auth supported = cephx 2026-02-22T19:51:52.408 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.408 INFO:teuthology.orchestra.run.trial017.stdout: filestore xattr use omap = true 2026-02-22T19:51:52.408 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.408 INFO:teuthology.orchestra.run.trial017.stdout: mon clock drift allowed = 1.000 2026-02-22T19:51:52.408 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.408 INFO:teuthology.orchestra.run.trial017.stdout: osd crush chooseleaf type = 0 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: auth debug = true 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: ms die on old message = true 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: ms die on bug = true 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: mon pg warn max object skew = 0 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.409 INFO:teuthology.orchestra.run.trial017.stdout: osd pool default size = 2 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: mon osd allow primary affinity = true 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: mon osd allow pg remap = true 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on legacy crush tunables = false 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on crush straw calc version zero = false 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on no sortbitwise = false 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on osd down out interval zero = false 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on too few osds = false 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: mon_allow_pool_size_one = true 2026-02-22T19:51:52.410 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.411 INFO:teuthology.orchestra.run.trial017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T19:51:52.411 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.411 INFO:teuthology.orchestra.run.trial017.stdout: osd default data pool replay window = 5 2026-02-22T19:51:52.411 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.411 INFO:teuthology.orchestra.run.trial017.stdout: mon allow pool delete = true 2026-02-22T19:51:52.411 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.411 INFO:teuthology.orchestra.run.trial017.stdout: mon cluster log file level = debug 2026-02-22T19:51:52.411 INFO:teuthology.orchestra.run.trial017.stdout: debug asserts on shutdown = true 2026-02-22T19:51:52.411 INFO:teuthology.orchestra.run.trial017.stdout: mon health detail to clog = false 2026-02-22T19:51:52.411 INFO:teuthology.orchestra.run.trial017.stdout: mon host = 10.20.193.17 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: mon client directed command retry = 5 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: mon warn on pool no app = False 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: ms inject socket failures = 5000 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: fsid = 10943a12-2d65-41e2-884e-f5f3b2ca910b 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout:[osd] 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: osd journal size = 100 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: osd scrub load threshold = 5.0 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: osd scrub max interval = 600 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: osd mclock profile = high_recovery_ops 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: osd mclock skip benchmark = true 2026-02-22T19:51:52.412 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.413 INFO:teuthology.orchestra.run.trial017.stdout: osd recover clone overlap = true 2026-02-22T19:51:52.413 INFO:teuthology.orchestra.run.trial017.stdout: osd recovery max chunk = 1048576 2026-02-22T19:51:52.413 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.413 INFO:teuthology.orchestra.run.trial017.stdout: osd debug shutdown = true 2026-02-22T19:51:52.413 INFO:teuthology.orchestra.run.trial017.stdout: osd debug op order = true 2026-02-22T19:51:52.413 INFO:teuthology.orchestra.run.trial017.stdout: osd debug verify stray on activate = true 2026-02-22T19:51:52.413 INFO:teuthology.orchestra.run.trial017.stdout: osd debug trim objects = true 2026-02-22T19:51:52.413 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.413 INFO:teuthology.orchestra.run.trial017.stdout: osd open classes on start = true 2026-02-22T19:51:52.413 INFO:teuthology.orchestra.run.trial017.stdout: osd debug pg log writeout = true 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: osd deep scrub update digest min age = 30 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: osd map max advance = 10 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: journal zero on create = true 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: filestore ondisk finisher threads = 3 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: filestore apply finisher threads = 3 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: bdev debug aio = true 2026-02-22T19:51:52.414 INFO:teuthology.orchestra.run.trial017.stdout: osd debug misdirected ops = true 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: bluestore block size = 96636764160 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: bluestore compression algorithm = zlib 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: bluestore compression mode = aggressive 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: bluestore fsck on mount = True 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: debug bluefs = 1/20 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: debug bluestore = 1/20 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: debug ms = 1 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: debug osd = 20 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: debug rocksdb = 4/10 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: mon osd full ratio = 0.9 2026-02-22T19:51:52.415 INFO:teuthology.orchestra.run.trial017.stdout: mon osd nearfull ratio = 0.8 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout: osd failsafe full ratio = 0.95 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout: osd objectstore = bluestore 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout:[mgr] 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout: debug ms = 1 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout: debug mgr = 20 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout: debug mon = 20 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout: debug auth = 20 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout: mgr/telemetry/nag = false 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.416 INFO:teuthology.orchestra.run.trial017.stdout:[mon] 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: debug ms = 1 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: debug mon = 20 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: debug paxos = 20 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: debug auth = 20 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: mon data avail warn = 5 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: mon mgr mkfs grace = 240 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: mon reweight min pgs per osd = 4 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: mon osd reporter subtree level = osd 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: mon osd prime pg temp = true 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: mon reweight min bytes per osd = 10 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.417 INFO:teuthology.orchestra.run.trial017.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: # 1m isn't quite enough 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: mon_down_mkfs_grace = 2m 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: mon_warn_on_filestore_osds = false 2026-02-22T19:51:52.418 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout:[client] 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout: rgw cache enabled = true 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout: rgw enable ops log = true 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout: rgw enable usage log = true 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout: debug rbd = 20 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout: debug rbd_mirror = 15 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout: log to stderr = False 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout: rbd default features = 125 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout:[mon.a] 2026-02-22T19:51:52.419 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.0] 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.1] 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.2] 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.3] 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.420 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.4] 2026-02-22T19:51:52.421 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.421 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.421 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.5] 2026-02-22T19:51:52.421 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.421 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.421 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.6] 2026-02-22T19:51:52.421 INFO:teuthology.orchestra.run.trial017.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-22T19:51:52.421 INFO:teuthology.orchestra.run.trial017.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-22T19:51:52.422 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2026-02-22T19:51:52.423 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2026-02-22T19:51:52.485 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-22T19:51:52.485 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:52.485 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2026-02-22T19:51:52.535 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:52.535 DEBUG:teuthology.orchestra.run.trial017:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2026-02-22T19:51:52.582 INFO:tasks.ceph:Sending monmap to node ubuntu@trial007.front.sepia.ceph.com 2026-02-22T19:51:52.583 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:52.583 DEBUG:teuthology.orchestra.run.trial007:> sudo dd of=/etc/ceph/cluster2.keyring 2026-02-22T19:51:52.583 DEBUG:teuthology.orchestra.run.trial007:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-22T19:51:52.597 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:51:52.597 DEBUG:teuthology.orchestra.run.trial007:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:52.644 INFO:tasks.ceph:Sending monmap to node ubuntu@trial017.front.sepia.ceph.com 2026-02-22T19:51:52.645 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:52.645 DEBUG:teuthology.orchestra.run.trial017:> sudo dd of=/etc/ceph/cluster2.keyring 2026-02-22T19:51:52.645 DEBUG:teuthology.orchestra.run.trial017:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-22T19:51:52.659 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:51:52.659 DEBUG:teuthology.orchestra.run.trial017:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:52.706 INFO:tasks.ceph:Setting up mon nodes... 2026-02-22T19:51:52.706 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-22T19:51:52.706 DEBUG:teuthology.orchestra.run.trial007:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2026-02-22T19:51:52.729 INFO:teuthology.orchestra.run.trial007.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2026-02-22T19:51:52.731 INFO:tasks.ceph:Setting up mds nodes... 2026-02-22T19:51:52.732 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-22T19:51:52.732 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2026-02-22T19:51:52.794 INFO:teuthology.orchestra.run.trial007.stdout:creating /etc/ceph/cluster2.client.0.keyring 2026-02-22T19:51:52.802 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2026-02-22T19:51:52.820 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2026-02-22T19:51:52.827 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2026-02-22T19:51:52.887 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2026-02-22T19:51:52.894 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2026-02-22T19:51:52.951 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2026-02-22T19:51:52.958 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2026-02-22T19:51:53.015 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2026-02-22T19:51:53.022 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2026-02-22T19:51:53.078 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2026-02-22T19:51:53.086 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2026-02-22T19:51:53.142 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2026-02-22T19:51:53.150 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2026-02-22T19:51:53.207 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2026-02-22T19:51:53.214 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2026-02-22T19:51:53.271 INFO:teuthology.orchestra.run.trial017.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2026-02-22T19:51:53.277 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-22T19:51:53.278 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial007.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@trial017.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T19:51:53.278 DEBUG:teuthology.orchestra.run.trial017:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2026-02-22T19:51:53.326 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:53.326 INFO:tasks.ceph:role: cluster2.osd.0 2026-02-22T19:51:53.326 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial017.front.sepia.ceph.com 2026-02-22T19:51:53.327 DEBUG:teuthology.orchestra.run.trial017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-22T19:51:53.375 INFO:teuthology.orchestra.run.trial017.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:53.376 INFO:teuthology.orchestra.run.trial017.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:53.376 INFO:teuthology.orchestra.run.trial017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:53.376 INFO:teuthology.orchestra.run.trial017.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:53.376 INFO:teuthology.orchestra.run.trial017.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:53.376 INFO:teuthology.orchestra.run.trial017.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:53.376 INFO:teuthology.orchestra.run.trial017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:53.376 INFO:teuthology.orchestra.run.trial017.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:53.376 INFO:teuthology.orchestra.run.trial017.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:53.376 INFO:teuthology.orchestra.run.trial017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:53.949 INFO:teuthology.orchestra.run.trial017.stdout:Discarding blocks...Done. 2026-02-22T19:51:53.951 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial017.front.sepia.ceph.com -o noatime 2026-02-22T19:51:53.951 DEBUG:teuthology.orchestra.run.trial017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2026-02-22T19:51:54.008 DEBUG:teuthology.orchestra.run.trial017:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2026-02-22T19:51:54.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:54.057 INFO:teuthology.orchestra.run.trial017.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:54.057 DEBUG:teuthology.orchestra.run.trial017:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2026-02-22T19:51:54.106 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:54.107 INFO:tasks.ceph:role: cluster2.osd.1 2026-02-22T19:51:54.107 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial017.front.sepia.ceph.com 2026-02-22T19:51:54.107 DEBUG:teuthology.orchestra.run.trial017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-22T19:51:54.160 INFO:teuthology.orchestra.run.trial017.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:54.161 INFO:teuthology.orchestra.run.trial017.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:54.161 INFO:teuthology.orchestra.run.trial017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:54.161 INFO:teuthology.orchestra.run.trial017.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:54.161 INFO:teuthology.orchestra.run.trial017.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:54.161 INFO:teuthology.orchestra.run.trial017.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:54.161 INFO:teuthology.orchestra.run.trial017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:54.161 INFO:teuthology.orchestra.run.trial017.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:54.161 INFO:teuthology.orchestra.run.trial017.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:54.161 INFO:teuthology.orchestra.run.trial017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:54.727 INFO:teuthology.orchestra.run.trial017.stdout:Discarding blocks...Done. 2026-02-22T19:51:54.729 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial017.front.sepia.ceph.com -o noatime 2026-02-22T19:51:54.729 DEBUG:teuthology.orchestra.run.trial017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2026-02-22T19:51:54.787 DEBUG:teuthology.orchestra.run.trial017:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2026-02-22T19:51:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:54.837 INFO:teuthology.orchestra.run.trial017.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:54.837 DEBUG:teuthology.orchestra.run.trial017:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2026-02-22T19:51:54.886 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T19:51:54.887 INFO:tasks.ceph:role: cluster2.osd.2 2026-02-22T19:51:54.887 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial017.front.sepia.ceph.com 2026-02-22T19:51:54.887 DEBUG:teuthology.orchestra.run.trial017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-22T19:51:54.939 INFO:teuthology.orchestra.run.trial017.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T19:51:54.940 INFO:teuthology.orchestra.run.trial017.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T19:51:54.940 INFO:teuthology.orchestra.run.trial017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T19:51:54.940 INFO:teuthology.orchestra.run.trial017.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T19:51:54.940 INFO:teuthology.orchestra.run.trial017.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T19:51:54.940 INFO:teuthology.orchestra.run.trial017.stdout: = sunit=0 swidth=0 blks 2026-02-22T19:51:54.940 INFO:teuthology.orchestra.run.trial017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T19:51:54.940 INFO:teuthology.orchestra.run.trial017.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T19:51:54.940 INFO:teuthology.orchestra.run.trial017.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T19:51:54.940 INFO:teuthology.orchestra.run.trial017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T19:51:55.501 INFO:teuthology.orchestra.run.trial017.stdout:Discarding blocks...Done. 2026-02-22T19:51:55.503 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial017.front.sepia.ceph.com -o noatime 2026-02-22T19:51:55.503 DEBUG:teuthology.orchestra.run.trial017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2026-02-22T19:51:55.563 DEBUG:teuthology.orchestra.run.trial017:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2026-02-22T19:51:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:51:55.613 INFO:teuthology.orchestra.run.trial017.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T19:51:55.613 DEBUG:teuthology.orchestra.run.trial017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:55.675 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:51:55.673+0000 7f87f31c58c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2026-02-22T19:51:55.676 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:51:55.673+0000 7f87f31c58c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2026-02-22T19:51:55.676 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:51:55.673+0000 7f87f31c58c0 -1 bdev(0x563bc7c5f800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2026-02-22T19:51:55.676 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:51:55.673+0000 7f87f31c58c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2026-02-22T19:51:58.802 DEBUG:teuthology.orchestra.run.trial017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2026-02-22T19:51:58.851 DEBUG:teuthology.orchestra.run.trial017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:51:58.915 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:51:58.913+0000 7f5f4f4978c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2026-02-22T19:51:58.916 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:51:58.913+0000 7f5f4f4978c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2026-02-22T19:51:58.916 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:51:58.913+0000 7f5f4f4978c0 -1 bdev(0x56149ff9b800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2026-02-22T19:51:58.916 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:51:58.913+0000 7f5f4f4978c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2026-02-22T19:52:02.006 DEBUG:teuthology.orchestra.run.trial017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2026-02-22T19:52:02.056 DEBUG:teuthology.orchestra.run.trial017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:52:02.119 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:52:02.117+0000 7f67dddfa8c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2026-02-22T19:52:02.120 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:52:02.117+0000 7f67dddfa8c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2026-02-22T19:52:02.120 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:52:02.117+0000 7f67dddfa8c0 -1 bdev(0x56441ad47800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2026-02-22T19:52:02.120 INFO:teuthology.orchestra.run.trial017.stderr:2026-02-22T19:52:02.117+0000 7f67dddfa8c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2026-02-22T19:52:05.274 DEBUG:teuthology.orchestra.run.trial017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2026-02-22T19:52:05.323 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-22T19:52:05.324 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:52:05.324 DEBUG:teuthology.orchestra.run.trial007:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2026-02-22T19:52:05.335 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:05.335 DEBUG:teuthology.orchestra.run.trial017:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2026-02-22T19:52:05.375 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:05.376 DEBUG:teuthology.orchestra.run.trial017:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2026-02-22T19:52:05.427 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:05.428 DEBUG:teuthology.orchestra.run.trial017:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2026-02-22T19:52:05.479 DEBUG:teuthology.orchestra.run.trial007:> set -ex 2026-02-22T19:52:05.480 DEBUG:teuthology.orchestra.run.trial007:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2026-02-22T19:52:05.486 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:05.486 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2026-02-22T19:52:05.526 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:05.527 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2026-02-22T19:52:05.571 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:05.571 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2026-02-22T19:52:05.619 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:05.619 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2026-02-22T19:52:05.667 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:05.667 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2026-02-22T19:52:05.716 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:05.716 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2026-02-22T19:52:05.762 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:05.762 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2026-02-22T19:52:05.807 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:05.807 DEBUG:teuthology.orchestra.run.trial017:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2026-02-22T19:52:05.854 INFO:tasks.ceph:Adding keys to all mons... 2026-02-22T19:52:05.855 DEBUG:teuthology.orchestra.run.trial017:> sudo tee -a /etc/ceph/cluster2.keyring 2026-02-22T19:52:05.902 INFO:teuthology.orchestra.run.trial017.stdout:[mgr.x] 2026-02-22T19:52:05.902 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBYXptppnddKxAACMRKHN+r6b3M+C7iAHiIPg== 2026-02-22T19:52:05.902 INFO:teuthology.orchestra.run.trial017.stdout:[osd.0] 2026-02-22T19:52:05.903 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBbXptpAxZMKBAAb50koB2IMoMwA63nLWgEYA== 2026-02-22T19:52:05.903 INFO:teuthology.orchestra.run.trial017.stdout:[osd.1] 2026-02-22T19:52:05.903 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBeXptpphOWNhAA56Oi4OwYvr92gCvOqxzwew== 2026-02-22T19:52:05.903 INFO:teuthology.orchestra.run.trial017.stdout:[osd.2] 2026-02-22T19:52:05.903 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBiXptpXjgkBxAAWOMhIFLaKefhr450HO/e2A== 2026-02-22T19:52:05.903 INFO:teuthology.orchestra.run.trial017.stdout:[client.0] 2026-02-22T19:52:05.903 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBYXptpRVs6LxAA0NLQu/o/CjuDfucLzyExMQ== 2026-02-22T19:52:05.903 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror] 2026-02-22T19:52:05.903 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBYXptpBVryMBAAcB7Hj+R9oo7GjhU/BCXhLA== 2026-02-22T19:52:05.903 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.0] 2026-02-22T19:52:05.903 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBYXptpyiHzNBAA8+tcX3POTdJefVg/SESDVQ== 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.1] 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBYXptpX7nFOBAAYRelmmdP5xhN9VFo7xRHsg== 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.2] 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBZXptpbS73ABAA8HefkQ39p8T3vOCW+GrNGg== 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.3] 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBZXptprCG7BBAAfqHj+Z+AWP8CbQgjCMqnIw== 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.4] 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBZXptpNeuLCBAAFON7i2sE/ExXObQnX63Rkg== 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.5] 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBZXptpRb9gDBAAnnzrAjsZS6VwFM6JqCyNng== 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.6] 2026-02-22T19:52:05.904 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBZXptpfXsxEBAAzUgbYRV0f8BDRkZ66f8Jjw== 2026-02-22T19:52:05.906 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T19:52:05.968 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:52:06.032 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:52:06.095 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T19:52:06.158 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:06.220 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:06.284 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:06.348 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:06.412 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:06.477 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:06.539 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:06.604 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:06.668 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T19:52:06.732 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-22T19:52:06.733 DEBUG:teuthology.orchestra.run.trial017:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2026-02-22T19:52:06.782 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2026-02-22T19:52:06.848 DEBUG:teuthology.orchestra.run.trial017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2026-02-22T19:52:06.898 DEBUG:teuthology.orchestra.run.trial017:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2026-02-22T19:52:06.946 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2026-02-22T19:52:06.946 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2026-02-22T19:52:06.947 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2026-02-22T19:52:06.988 INFO:tasks.ceph.cluster2.mon.a:Started 2026-02-22T19:52:06.988 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2026-02-22T19:52:06.988 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2026-02-22T19:52:06.988 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2026-02-22T19:52:06.991 INFO:tasks.ceph.cluster2.mgr.x:Started 2026-02-22T19:52:06.991 INFO:tasks.ceph:Setting crush tunables to default 2026-02-22T19:52:06.991 DEBUG:teuthology.orchestra.run.trial017:> sudo ceph --cluster cluster2 osd crush tunables default 2026-02-22T19:52:07.157 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.120+0000 7f5418785280 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-22T19:52:07.157 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.144+0000 7f5418785280 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-22T19:52:07.158 INFO:teuthology.orchestra.run.trial017.stderr:adjusted tunables profile to default 2026-02-22T19:52:07.158 INFO:tasks.ceph:check_enable_crimson: False 2026-02-22T19:52:07.158 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2026-02-22T19:52:07.158 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:07.158 DEBUG:teuthology.orchestra.run.trial017:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2026-02-22T19:52:07.167 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:07.167 DEBUG:teuthology.orchestra.run.trial017:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2026-02-22T19:52:07.187 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.180+0000 7f5418785280 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-22T19:52:07.220 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:07.220 DEBUG:teuthology.orchestra.run.trial017:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2026-02-22T19:52:07.251 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.244+0000 7f5418785280 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-22T19:52:07.275 DEBUG:teuthology.orchestra.run.trial017:> sudo ceph --cluster cluster2 osd new bb49012e-bab3-4325-b6fa-8bb3ea30380c 0 2026-02-22T19:52:07.316 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.312+0000 7f5418785280 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-22T19:52:07.353 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.348+0000 7f5418785280 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-22T19:52:07.415 INFO:teuthology.orchestra.run.trial017.stdout:0 2026-02-22T19:52:07.420 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.416+0000 7f5418785280 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-22T19:52:07.424 DEBUG:teuthology.orchestra.run.trial017:> sudo ceph --cluster cluster2 osd new 090c5ffe-2b5d-4ef9-96b9-a5a6e6232d90 1 2026-02-22T19:52:07.518 INFO:teuthology.orchestra.run.trial017.stdout:1 2026-02-22T19:52:07.527 DEBUG:teuthology.orchestra.run.trial017:> sudo ceph --cluster cluster2 osd new e43281b8-07e8-4d2b-90da-609ff2835583 2 2026-02-22T19:52:07.583 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.576+0000 7f5418785280 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-22T19:52:07.621 INFO:teuthology.orchestra.run.trial017.stdout:2 2026-02-22T19:52:07.630 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2026-02-22T19:52:07.631 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2026-02-22T19:52:07.632 INFO:tasks.ceph.cluster2.osd.0:Started 2026-02-22T19:52:07.633 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2026-02-22T19:52:07.633 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2026-02-22T19:52:07.635 INFO:tasks.ceph.cluster2.osd.1:Started 2026-02-22T19:52:07.635 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2026-02-22T19:52:07.635 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2026-02-22T19:52:07.637 INFO:tasks.ceph.cluster2.osd.2:Started 2026-02-22T19:52:07.637 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:07.741 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:52:07.742 INFO:teuthology.orchestra.run.trial017.stdout:{"epoch":5,"fsid":"10943a12-2d65-41e2-884e-f5f3b2ca910b","created":"2026-02-22T19:52:07.064998+0000","modified":"2026-02-22T19:52:07.621477+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:52:07.621477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bb49012e-bab3-4325-b6fa-8bb3ea30380c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"090c5ffe-2b5d-4ef9-96b9-a5a6e6232d90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e43281b8-07e8-4d2b-90da-609ff2835583","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:07.753 INFO:tasks.ceph.ceph_manager.cluster2:[] 2026-02-22T19:52:07.753 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-22T19:52:07.822 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.816+0000 7f5418785280 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-22T19:52:07.910 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.904+0000 7f5418785280 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-22T19:52:07.935 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.928+0000 7f5418785280 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-22T19:52:07.963 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.956+0000 7f5418785280 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-22T19:52:07.993 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:07.988+0000 7f5418785280 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-22T19:52:08.055 DEBUG:teuthology.orchestra.run.trial017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:08.088 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:08.080+0000 7f5418785280 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-22T19:52:08.144 INFO:teuthology.misc.health.trial017.stdout: 2026-02-22T19:52:08.145 INFO:teuthology.misc.health.trial017.stdout:{"epoch":5,"fsid":"10943a12-2d65-41e2-884e-f5f3b2ca910b","created":"2026-02-22T19:52:07.064998+0000","modified":"2026-02-22T19:52:07.621477+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:52:07.621477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bb49012e-bab3-4325-b6fa-8bb3ea30380c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"090c5ffe-2b5d-4ef9-96b9-a5a6e6232d90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e43281b8-07e8-4d2b-90da-609ff2835583","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:08.153 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T19:52:08.256 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-22T19:52:08.404 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:08.396+0000 7f5418785280 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-22T19:52:08.435 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:08.428+0000 7f5418785280 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-22T19:52:08.508 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:08.500+0000 7f5418785280 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-22T19:52:08.532 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:08.528+0000 7f5418785280 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-22T19:52:08.557 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:08.552+0000 7f5418785280 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-22T19:52:08.583 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:08.576+0000 7f5418785280 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-22T19:52:08.660 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-22T19:52:08.661 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-22T19:52:08.661 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr: from numpy import show_config as show_numpy_config 2026-02-22T19:52:08.667 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:08.660+0000 7f5418785280 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-22T19:52:08.700 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:08.692+0000 7f5418785280 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-22T19:52:08.737 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:08.732+0000 7f5418785280 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-22T19:52:09.176 INFO:tasks.ceph.cluster2.osd.0.trial017.stderr:2026-02-22T19:52:09.173+0000 7f6b5669c8c0 -1 Falling back to public interface 2026-02-22T19:52:09.200 INFO:tasks.ceph.cluster2.osd.2.trial017.stderr:2026-02-22T19:52:09.197+0000 7fcdc0a118c0 -1 Falling back to public interface 2026-02-22T19:52:09.207 INFO:tasks.ceph.cluster2.osd.1.trial017.stderr:2026-02-22T19:52:09.205+0000 7f0988b668c0 -1 Falling back to public interface 2026-02-22T19:52:09.746 INFO:tasks.ceph.cluster2.mgr.x.trial007.stderr:2026-02-22T19:52:09.740+0000 7f5336fd1640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-22T19:52:12.965 INFO:tasks.ceph.cluster2.osd.0.trial017.stderr:2026-02-22T19:52:12.961+0000 7f6b5669c8c0 -1 osd.0 0 log_to_monitors true 2026-02-22T19:52:13.109 INFO:tasks.ceph.cluster2.osd.1.trial017.stderr:2026-02-22T19:52:13.105+0000 7f0988b668c0 -1 osd.1 0 log_to_monitors true 2026-02-22T19:52:13.540 INFO:tasks.ceph.cluster2.osd.2.trial017.stderr:2026-02-22T19:52:13.537+0000 7fcdc0a118c0 -1 osd.2 0 log_to_monitors true 2026-02-22T19:52:14.456 DEBUG:teuthology.orchestra.run.trial017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:14.601 INFO:teuthology.misc.health.trial017.stdout: 2026-02-22T19:52:14.602 INFO:teuthology.misc.health.trial017.stdout:{"epoch":6,"fsid":"10943a12-2d65-41e2-884e-f5f3b2ca910b","created":"2026-02-22T19:52:07.064998+0000","modified":"2026-02-22T19:52:13.745781+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T19:52:07.621477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bb49012e-bab3-4325-b6fa-8bb3ea30380c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"090c5ffe-2b5d-4ef9-96b9-a5a6e6232d90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e43281b8-07e8-4d2b-90da-609ff2835583","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:14.611 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T19:52:14.754 INFO:tasks.ceph.cluster2.osd.2.trial017.stderr:2026-02-22T19:52:14.749+0000 7fcdbc9ba640 -1 osd.2 0 waiting for initial osdmap 2026-02-22T19:52:14.755 INFO:tasks.ceph.cluster2.osd.0.trial017.stderr:2026-02-22T19:52:14.749+0000 7f6b52645640 -1 osd.0 0 waiting for initial osdmap 2026-02-22T19:52:14.755 INFO:tasks.ceph.cluster2.osd.1.trial017.stderr:2026-02-22T19:52:14.753+0000 7f0984b0f640 -1 osd.1 0 waiting for initial osdmap 2026-02-22T19:52:14.776 INFO:tasks.ceph.cluster2.osd.0.trial017.stderr:2026-02-22T19:52:14.773+0000 7f6b4a478640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:52:14.776 INFO:tasks.ceph.cluster2.osd.1.trial017.stderr:2026-02-22T19:52:14.773+0000 7f097c942640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:52:14.777 INFO:tasks.ceph.cluster2.osd.2.trial017.stderr:2026-02-22T19:52:14.773+0000 7fcdb47ed640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T19:52:20.915 DEBUG:teuthology.orchestra.run.trial017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:21.055 INFO:teuthology.misc.health.trial017.stdout: 2026-02-22T19:52:21.056 INFO:teuthology.misc.health.trial017.stdout:{"epoch":12,"fsid":"10943a12-2d65-41e2-884e-f5f3b2ca910b","created":"2026-02-22T19:52:07.064998+0000","modified":"2026-02-22T19:52:19.760710+0000","last_up_change":"2026-02-22T19:52:15.751975+0000","last_in_change":"2026-02-22T19:52:07.621477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:52:16.778879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bb49012e-bab3-4325-b6fa-8bb3ea30380c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6800","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6801","nonce":2343458682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6802","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6803","nonce":2343458682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6806","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6807","nonce":2343458682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6804","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6805","nonce":2343458682}]},"public_addr":"10.20.193.17:6801/2343458682","cluster_addr":"10.20.193.17:6803/2343458682","heartbeat_back_addr":"10.20.193.17:6807/2343458682","heartbeat_front_addr":"10.20.193.17:6805/2343458682","state":["exists","up"]},{"osd":1,"uuid":"090c5ffe-2b5d-4ef9-96b9-a5a6e6232d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6816","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6817","nonce":92566604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6818","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6819","nonce":92566604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6822","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6823","nonce":92566604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6820","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6821","nonce":92566604}]},"public_addr":"10.20.193.17:6817/92566604","cluster_addr":"10.20.193.17:6819/92566604","heartbeat_back_addr":"10.20.193.17:6823/92566604","heartbeat_front_addr":"10.20.193.17:6821/92566604","state":["exists","up"]},{"osd":2,"uuid":"e43281b8-07e8-4d2b-90da-609ff2835583","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6808","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6809","nonce":2835130688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6810","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6811","nonce":2835130688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6814","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6815","nonce":2835130688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6812","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6813","nonce":2835130688}]},"public_addr":"10.20.193.17:6809/2835130688","cluster_addr":"10.20.193.17:6811/2835130688","heartbeat_back_addr":"10.20.193.17:6815/2835130688","heartbeat_front_addr":"10.20.193.17:6813/2835130688","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:13.926503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:14.155040+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:14.538654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:21.065 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-22T19:52:21.065 INFO:tasks.ceph:Creating RBD pool 2026-02-22T19:52:21.065 DEBUG:teuthology.orchestra.run.trial017:> sudo ceph --cluster cluster2 osd pool create rbd 8 2026-02-22T19:52:21.768 INFO:teuthology.orchestra.run.trial017.stderr:pool 'rbd' created 2026-02-22T19:52:21.780 DEBUG:teuthology.orchestra.run.trial017:> rbd --cluster cluster2 pool init rbd 2026-02-22T19:52:24.783 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2026-02-22T19:52:24.784 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2026-02-22T19:52:24.784 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2026-02-22T19:52:24.784 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2026-02-22T19:52:24.784 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-22T19:52:24.950 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:52:24.960 INFO:teuthology.orchestra.run.trial017.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6826","nonce":1236559326},{"type":"v1","addr":"10.20.193.7:6827","nonce":1236559326}]},"active_addr":"10.20.193.7:6827/1236559326","active_change":"2026-02-22T19:52:08.739812+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.7:0","nonce":2372450603}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.7:0","nonce":3869643582}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.7:0","nonce":3620412440}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.7:0","nonce":1814418298}]}]} 2026-02-22T19:52:24.963 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2026-02-22T19:52:24.963 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2026-02-22T19:52:24.963 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:25.106 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:52:25.107 INFO:teuthology.orchestra.run.trial017.stdout:{"epoch":16,"fsid":"10943a12-2d65-41e2-884e-f5f3b2ca910b","created":"2026-02-22T19:52:07.064998+0000","modified":"2026-02-22T19:52:24.774008+0000","last_up_change":"2026-02-22T19:52:15.751975+0000","last_in_change":"2026-02-22T19:52:07.621477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:52:16.778879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:52:21.214161+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bb49012e-bab3-4325-b6fa-8bb3ea30380c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6800","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6801","nonce":2343458682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6802","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6803","nonce":2343458682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6806","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6807","nonce":2343458682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6804","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6805","nonce":2343458682}]},"public_addr":"10.20.193.17:6801/2343458682","cluster_addr":"10.20.193.17:6803/2343458682","heartbeat_back_addr":"10.20.193.17:6807/2343458682","heartbeat_front_addr":"10.20.193.17:6805/2343458682","state":["exists","up"]},{"osd":1,"uuid":"090c5ffe-2b5d-4ef9-96b9-a5a6e6232d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6816","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6817","nonce":92566604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6818","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6819","nonce":92566604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6822","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6823","nonce":92566604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6820","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6821","nonce":92566604}]},"public_addr":"10.20.193.17:6817/92566604","cluster_addr":"10.20.193.17:6819/92566604","heartbeat_back_addr":"10.20.193.17:6823/92566604","heartbeat_front_addr":"10.20.193.17:6821/92566604","state":["exists","up"]},{"osd":2,"uuid":"e43281b8-07e8-4d2b-90da-609ff2835583","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6808","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6809","nonce":2835130688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6810","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6811","nonce":2835130688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6814","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6815","nonce":2835130688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6812","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6813","nonce":2835130688}]},"public_addr":"10.20.193.17:6809/2835130688","cluster_addr":"10.20.193.17:6811/2835130688","heartbeat_back_addr":"10.20.193.17:6815/2835130688","heartbeat_front_addr":"10.20.193.17:6813/2835130688","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:13.926503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:14.155040+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:14.538654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:25.117 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2026-02-22T19:52:25.117 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-22T19:52:25.259 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:52:25.260 INFO:teuthology.orchestra.run.trial017.stdout:{"epoch":16,"fsid":"10943a12-2d65-41e2-884e-f5f3b2ca910b","created":"2026-02-22T19:52:07.064998+0000","modified":"2026-02-22T19:52:24.774008+0000","last_up_change":"2026-02-22T19:52:15.751975+0000","last_in_change":"2026-02-22T19:52:07.621477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:52:16.778879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:52:21.214161+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bb49012e-bab3-4325-b6fa-8bb3ea30380c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6800","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6801","nonce":2343458682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6802","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6803","nonce":2343458682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6806","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6807","nonce":2343458682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6804","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6805","nonce":2343458682}]},"public_addr":"10.20.193.17:6801/2343458682","cluster_addr":"10.20.193.17:6803/2343458682","heartbeat_back_addr":"10.20.193.17:6807/2343458682","heartbeat_front_addr":"10.20.193.17:6805/2343458682","state":["exists","up"]},{"osd":1,"uuid":"090c5ffe-2b5d-4ef9-96b9-a5a6e6232d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6816","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6817","nonce":92566604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6818","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6819","nonce":92566604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6822","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6823","nonce":92566604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6820","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6821","nonce":92566604}]},"public_addr":"10.20.193.17:6817/92566604","cluster_addr":"10.20.193.17:6819/92566604","heartbeat_back_addr":"10.20.193.17:6823/92566604","heartbeat_front_addr":"10.20.193.17:6821/92566604","state":["exists","up"]},{"osd":2,"uuid":"e43281b8-07e8-4d2b-90da-609ff2835583","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6808","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6809","nonce":2835130688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6810","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6811","nonce":2835130688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6814","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6815","nonce":2835130688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6812","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6813","nonce":2835130688}]},"public_addr":"10.20.193.17:6809/2835130688","cluster_addr":"10.20.193.17:6811/2835130688","heartbeat_back_addr":"10.20.193.17:6815/2835130688","heartbeat_front_addr":"10.20.193.17:6813/2835130688","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:13.926503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:14.155040+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:14.538654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T19:52:25.271 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2026-02-22T19:52:25.271 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2026-02-22T19:52:25.272 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2026-02-22T19:52:25.355 INFO:teuthology.orchestra.run.trial017.stdout:34359738371 2026-02-22T19:52:25.356 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2026-02-22T19:52:25.357 INFO:teuthology.orchestra.run.trial017.stdout:34359738372 2026-02-22T19:52:25.358 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2026-02-22T19:52:25.358 INFO:teuthology.orchestra.run.trial017.stdout:34359738371 2026-02-22T19:52:25.359 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2026-02-22T19:52:25.547 INFO:teuthology.orchestra.run.trial017.stdout:34359738370 2026-02-22T19:52:25.554 INFO:teuthology.orchestra.run.trial017.stdout:34359738371 2026-02-22T19:52:25.554 INFO:teuthology.orchestra.run.trial017.stdout:34359738370 2026-02-22T19:52:25.559 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.0 2026-02-22T19:52:25.565 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.2 2026-02-22T19:52:25.567 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.1 2026-02-22T19:52:26.559 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2026-02-22T19:52:26.566 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2026-02-22T19:52:26.567 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2026-02-22T19:52:26.707 INFO:teuthology.orchestra.run.trial017.stdout:34359738370 2026-02-22T19:52:26.719 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.0 2026-02-22T19:52:26.746 INFO:teuthology.orchestra.run.trial017.stdout:34359738371 2026-02-22T19:52:26.749 INFO:teuthology.orchestra.run.trial017.stdout:34359738370 2026-02-22T19:52:26.757 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.2 2026-02-22T19:52:26.761 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.1 2026-02-22T19:52:27.720 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2026-02-22T19:52:27.758 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2026-02-22T19:52:27.762 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2026-02-22T19:52:27.870 INFO:teuthology.orchestra.run.trial017.stdout:34359738371 2026-02-22T19:52:27.882 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.0 2026-02-22T19:52:27.883 DEBUG:teuthology.parallel:result is None 2026-02-22T19:52:27.911 INFO:teuthology.orchestra.run.trial017.stdout:34359738371 2026-02-22T19:52:27.915 INFO:teuthology.orchestra.run.trial017.stdout:34359738372 2026-02-22T19:52:27.922 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.1 2026-02-22T19:52:27.922 DEBUG:teuthology.parallel:result is None 2026-02-22T19:52:27.927 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.2 2026-02-22T19:52:27.927 DEBUG:teuthology.parallel:result is None 2026-02-22T19:52:27.927 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2026-02-22T19:52:27.927 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T19:52:28.115 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:52:28.115 INFO:teuthology.orchestra.run.trial017.stderr:dumped all 2026-02-22T19:52:28.127 INFO:teuthology.orchestra.run.trial017.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2026-02-22T19:52:26.742140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81428,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034092,"statfs":{"total":289910292480,"available":289826910208,"internally_reserved":0,"allocated":532480,"data_stored":1299536,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.975904"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.779439+0000","last_change":"2026-02-22T19:52:24.779439+0000","last_active":"2026-02-22T19:52:24.779439+0000","last_peered":"2026-02-22T19:52:24.779439+0000","last_clean":"2026-02-22T19:52:24.779439+0000","last_became_active":"2026-02-22T19:52:22.776103+0000","last_became_peered":"2026-02-22T19:52:22.776103+0000","last_unstale":"2026-02-22T19:52:24.779439+0000","last_undegraded":"2026-02-22T19:52:24.779439+0000","last_fullsized":"2026-02-22T19:52:24.779439+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:02:03.695406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044655200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.779510+0000","last_change":"2026-02-22T19:52:24.779510+0000","last_active":"2026-02-22T19:52:24.779510+0000","last_peered":"2026-02-22T19:52:24.779510+0000","last_clean":"2026-02-22T19:52:24.779510+0000","last_became_active":"2026-02-22T19:52:22.776089+0000","last_became_peered":"2026-02-22T19:52:22.776089+0000","last_unstale":"2026-02-22T19:52:24.779510+0000","last_undegraded":"2026-02-22T19:52:24.779510+0000","last_fullsized":"2026-02-22T19:52:24.779510+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:33:54.099391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051971799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.779619+0000","last_change":"2026-02-22T19:52:24.779619+0000","last_active":"2026-02-22T19:52:24.779619+0000","last_peered":"2026-02-22T19:52:24.779619+0000","last_clean":"2026-02-22T19:52:24.779619+0000","last_became_active":"2026-02-22T19:52:22.775967+0000","last_became_peered":"2026-02-22T19:52:22.775967+0000","last_unstale":"2026-02-22T19:52:24.779619+0000","last_undegraded":"2026-02-22T19:52:24.779619+0000","last_fullsized":"2026-02-22T19:52:24.779619+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:57:59.097058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060316299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.779289+0000","last_change":"2026-02-22T19:52:24.779289+0000","last_active":"2026-02-22T19:52:24.779289+0000","last_peered":"2026-02-22T19:52:24.779289+0000","last_clean":"2026-02-22T19:52:24.779289+0000","last_became_active":"2026-02-22T19:52:22.775149+0000","last_became_peered":"2026-02-22T19:52:22.775149+0000","last_unstale":"2026-02-22T19:52:24.779289+0000","last_undegraded":"2026-02-22T19:52:24.779289+0000","last_fullsized":"2026-02-22T19:52:24.779289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:14:32.713489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033844000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.778509+0000","last_change":"2026-02-22T19:52:24.777066+0000","last_active":"2026-02-22T19:52:24.778509+0000","last_peered":"2026-02-22T19:52:24.778509+0000","last_clean":"2026-02-22T19:52:24.778509+0000","last_became_active":"2026-02-22T19:52:22.775845+0000","last_became_peered":"2026-02-22T19:52:22.775845+0000","last_unstale":"2026-02-22T19:52:24.778509+0000","last_undegraded":"2026-02-22T19:52:24.778509+0000","last_fullsized":"2026-02-22T19:52:24.778509+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:44:39.886032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033930199999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:52:23.773690+0000","last_change":"2026-02-22T19:52:22.776155+0000","last_active":"2026-02-22T19:52:23.773690+0000","last_peered":"2026-02-22T19:52:23.773690+0000","last_clean":"2026-02-22T19:52:23.773690+0000","last_became_active":"2026-02-22T19:52:22.775858+0000","last_became_peered":"2026-02-22T19:52:22.775858+0000","last_unstale":"2026-02-22T19:52:23.773690+0000","last_undegraded":"2026-02-22T19:52:23.773690+0000","last_fullsized":"2026-02-22T19:52:23.773690+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:47:06.163864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:52:23.773675+0000","last_change":"2026-02-22T19:52:22.775800+0000","last_active":"2026-02-22T19:52:23.773675+0000","last_peered":"2026-02-22T19:52:23.773675+0000","last_clean":"2026-02-22T19:52:23.773675+0000","last_became_active":"2026-02-22T19:52:22.775507+0000","last_became_peered":"2026-02-22T19:52:22.775507+0000","last_unstale":"2026-02-22T19:52:23.773675+0000","last_undegraded":"2026-02-22T19:52:23.773675+0000","last_fullsized":"2026-02-22T19:52:23.773675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:41:12.144464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.779287+0000","last_change":"2026-02-22T19:52:24.779287+0000","last_active":"2026-02-22T19:52:24.779287+0000","last_peered":"2026-02-22T19:52:24.779287+0000","last_clean":"2026-02-22T19:52:24.779287+0000","last_became_active":"2026-02-22T19:52:22.774567+0000","last_became_peered":"2026-02-22T19:52:22.774567+0000","last_unstale":"2026-02-22T19:52:24.779287+0000","last_undegraded":"2026-02-22T19:52:24.779287+0000","last_fullsized":"2026-02-22T19:52:24.779287+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:41:09.322918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036364699999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'72","reported_seq":110,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.778850+0000","last_change":"2026-02-22T19:52:18.864376+0000","last_active":"2026-02-22T19:52:24.778850+0000","last_peered":"2026-02-22T19:52:24.778850+0000","last_clean":"2026-02-22T19:52:24.778850+0000","last_became_active":"2026-02-22T19:52:18.863570+0000","last_became_peered":"2026-02-22T19:52:18.863570+0000","last_unstale":"2026-02-22T19:52:24.778850+0000","last_undegraded":"2026-02-22T19:52:24.778850+0000","last_fullsized":"2026-02-22T19:52:24.778850+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_clean_scrub_stamp":"2026-02-22T19:52:17.758264+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:09:52.495537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":200704,"data_stored":631161,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":200704,"data_stored":631161,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27096,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344744,"statfs":{"total":96636764160,"available":96609017856,"internally_reserved":0,"allocated":131072,"data_stored":37214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T19:52:28.129 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T19:52:28.283 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:52:28.284 INFO:teuthology.orchestra.run.trial017.stderr:dumped all 2026-02-22T19:52:28.294 INFO:teuthology.orchestra.run.trial017.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2026-02-22T19:52:26.742140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81428,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034092,"statfs":{"total":289910292480,"available":289826910208,"internally_reserved":0,"allocated":532480,"data_stored":1299536,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.975904"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.779439+0000","last_change":"2026-02-22T19:52:24.779439+0000","last_active":"2026-02-22T19:52:24.779439+0000","last_peered":"2026-02-22T19:52:24.779439+0000","last_clean":"2026-02-22T19:52:24.779439+0000","last_became_active":"2026-02-22T19:52:22.776103+0000","last_became_peered":"2026-02-22T19:52:22.776103+0000","last_unstale":"2026-02-22T19:52:24.779439+0000","last_undegraded":"2026-02-22T19:52:24.779439+0000","last_fullsized":"2026-02-22T19:52:24.779439+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:02:03.695406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044655200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.779510+0000","last_change":"2026-02-22T19:52:24.779510+0000","last_active":"2026-02-22T19:52:24.779510+0000","last_peered":"2026-02-22T19:52:24.779510+0000","last_clean":"2026-02-22T19:52:24.779510+0000","last_became_active":"2026-02-22T19:52:22.776089+0000","last_became_peered":"2026-02-22T19:52:22.776089+0000","last_unstale":"2026-02-22T19:52:24.779510+0000","last_undegraded":"2026-02-22T19:52:24.779510+0000","last_fullsized":"2026-02-22T19:52:24.779510+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:33:54.099391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051971799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.779619+0000","last_change":"2026-02-22T19:52:24.779619+0000","last_active":"2026-02-22T19:52:24.779619+0000","last_peered":"2026-02-22T19:52:24.779619+0000","last_clean":"2026-02-22T19:52:24.779619+0000","last_became_active":"2026-02-22T19:52:22.775967+0000","last_became_peered":"2026-02-22T19:52:22.775967+0000","last_unstale":"2026-02-22T19:52:24.779619+0000","last_undegraded":"2026-02-22T19:52:24.779619+0000","last_fullsized":"2026-02-22T19:52:24.779619+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:57:59.097058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060316299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.779289+0000","last_change":"2026-02-22T19:52:24.779289+0000","last_active":"2026-02-22T19:52:24.779289+0000","last_peered":"2026-02-22T19:52:24.779289+0000","last_clean":"2026-02-22T19:52:24.779289+0000","last_became_active":"2026-02-22T19:52:22.775149+0000","last_became_peered":"2026-02-22T19:52:22.775149+0000","last_unstale":"2026-02-22T19:52:24.779289+0000","last_undegraded":"2026-02-22T19:52:24.779289+0000","last_fullsized":"2026-02-22T19:52:24.779289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:14:32.713489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033844000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.778509+0000","last_change":"2026-02-22T19:52:24.777066+0000","last_active":"2026-02-22T19:52:24.778509+0000","last_peered":"2026-02-22T19:52:24.778509+0000","last_clean":"2026-02-22T19:52:24.778509+0000","last_became_active":"2026-02-22T19:52:22.775845+0000","last_became_peered":"2026-02-22T19:52:22.775845+0000","last_unstale":"2026-02-22T19:52:24.778509+0000","last_undegraded":"2026-02-22T19:52:24.778509+0000","last_fullsized":"2026-02-22T19:52:24.778509+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:44:39.886032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033930199999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:52:23.773690+0000","last_change":"2026-02-22T19:52:22.776155+0000","last_active":"2026-02-22T19:52:23.773690+0000","last_peered":"2026-02-22T19:52:23.773690+0000","last_clean":"2026-02-22T19:52:23.773690+0000","last_became_active":"2026-02-22T19:52:22.775858+0000","last_became_peered":"2026-02-22T19:52:22.775858+0000","last_unstale":"2026-02-22T19:52:23.773690+0000","last_undegraded":"2026-02-22T19:52:23.773690+0000","last_fullsized":"2026-02-22T19:52:23.773690+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:47:06.163864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-22T19:52:23.773675+0000","last_change":"2026-02-22T19:52:22.775800+0000","last_active":"2026-02-22T19:52:23.773675+0000","last_peered":"2026-02-22T19:52:23.773675+0000","last_clean":"2026-02-22T19:52:23.773675+0000","last_became_active":"2026-02-22T19:52:22.775507+0000","last_became_peered":"2026-02-22T19:52:22.775507+0000","last_unstale":"2026-02-22T19:52:23.773675+0000","last_undegraded":"2026-02-22T19:52:23.773675+0000","last_fullsized":"2026-02-22T19:52:23.773675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:41:12.144464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.779287+0000","last_change":"2026-02-22T19:52:24.779287+0000","last_active":"2026-02-22T19:52:24.779287+0000","last_peered":"2026-02-22T19:52:24.779287+0000","last_clean":"2026-02-22T19:52:24.779287+0000","last_became_active":"2026-02-22T19:52:22.774567+0000","last_became_peered":"2026-02-22T19:52:22.774567+0000","last_unstale":"2026-02-22T19:52:24.779287+0000","last_undegraded":"2026-02-22T19:52:24.779287+0000","last_fullsized":"2026-02-22T19:52:24.779287+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:41:09.322918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036364699999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'72","reported_seq":110,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-22T19:52:24.778850+0000","last_change":"2026-02-22T19:52:18.864376+0000","last_active":"2026-02-22T19:52:24.778850+0000","last_peered":"2026-02-22T19:52:24.778850+0000","last_clean":"2026-02-22T19:52:24.778850+0000","last_became_active":"2026-02-22T19:52:18.863570+0000","last_became_peered":"2026-02-22T19:52:18.863570+0000","last_unstale":"2026-02-22T19:52:24.778850+0000","last_undegraded":"2026-02-22T19:52:24.778850+0000","last_fullsized":"2026-02-22T19:52:24.778850+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_clean_scrub_stamp":"2026-02-22T19:52:17.758264+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:09:52.495537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":200704,"data_stored":631161,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":200704,"data_stored":631161,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27096,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344744,"statfs":{"total":96636764160,"available":96609017856,"internally_reserved":0,"allocated":131072,"data_stored":37214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T19:52:28.295 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2026-02-22T19:52:28.295 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2026-02-22T19:52:28.295 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2026-02-22T19:52:28.296 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2026-02-22T19:52:28.465 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T19:52:28.465 INFO:teuthology.orchestra.run.trial017.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-22T19:52:28.475 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2026-02-22T19:52:28.476 INFO:teuthology.run_tasks:Running task exec... 2026-02-22T19:52:28.492 INFO:teuthology.task.exec:Executing custom commands... 2026-02-22T19:52:28.492 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@trial017.front.sepia.ceph.com 2026-02-22T19:52:28.492 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:28.676 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.4] 2026-02-22T19:52:28.676 INFO:teuthology.orchestra.run.trial017.stdout: key = AQAzXptp0ojrHBAAHx2/fY2te7+QJNjJahM+1A== 2026-02-22T19:52:28.676 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:28.676 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:28.676 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.4 2026-02-22T19:52:28.687 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:28.861 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.5] 2026-02-22T19:52:28.861 INFO:teuthology.orchestra.run.trial017.stdout: key = AQAzXptpi6E4IRAAPjW3HNUm1v7vuX2fd31Gdw== 2026-02-22T19:52:28.861 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:28.861 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:28.861 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.5 2026-02-22T19:52:28.872 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:29.054 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.6] 2026-02-22T19:52:29.055 INFO:teuthology.orchestra.run.trial017.stdout: key = AQAzXptpIKSCJRAAYCbHRbh9rZvbFknjkYyaVA== 2026-02-22T19:52:29.055 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:29.055 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:29.055 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.6 2026-02-22T19:52:29.066 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:29.242 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.4] 2026-02-22T19:52:29.242 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBZXptpNeuLCBAAFON7i2sE/ExXObQnX63Rkg== 2026-02-22T19:52:29.242 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:29.242 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:29.243 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.4 2026-02-22T19:52:29.253 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:29.436 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.5] 2026-02-22T19:52:29.436 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBZXptpRb9gDBAAnnzrAjsZS6VwFM6JqCyNng== 2026-02-22T19:52:29.437 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:29.437 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:29.437 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.5 2026-02-22T19:52:29.449 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:29.629 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.6] 2026-02-22T19:52:29.630 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBZXptpfXsxEBAAzUgbYRV0f8BDRkZ66f8Jjw== 2026-02-22T19:52:29.630 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:29.630 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:29.630 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.6 2026-02-22T19:52:29.642 INFO:teuthology.run_tasks:Running task exec... 2026-02-22T19:52:29.659 INFO:teuthology.task.exec:Executing custom commands... 2026-02-22T19:52:29.659 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@trial017.front.sepia.ceph.com 2026-02-22T19:52:29.659 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:29.839 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror] 2026-02-22T19:52:29.839 INFO:teuthology.orchestra.run.trial017.stdout: key = AQAzXptp2kIlBxAAZ+Y9S5ZbObyO8e1G5Zopng== 2026-02-22T19:52:29.839 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror-peer" 2026-02-22T19:52:29.839 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:29.840 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror 2026-02-22T19:52:29.850 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:30.030 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.0] 2026-02-22T19:52:30.030 INFO:teuthology.orchestra.run.trial017.stdout: key = AQAzXptp2MmRCxAAaTdk7uLqM5GHdCYSPqygXw== 2026-02-22T19:52:30.030 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:30.030 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:30.031 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.0 2026-02-22T19:52:30.041 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:30.218 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.1] 2026-02-22T19:52:30.218 INFO:teuthology.orchestra.run.trial017.stdout: key = AQAzXptpHY3VDxAAOovzBkM8iTL+qviosOyUKw== 2026-02-22T19:52:30.218 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:30.218 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:30.219 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.1 2026-02-22T19:52:30.229 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:30.412 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.2] 2026-02-22T19:52:30.412 INFO:teuthology.orchestra.run.trial017.stdout: key = AQAzXptpTuIaFBAAWgJJ82X1g9CQ3fkioHFLyA== 2026-02-22T19:52:30.412 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:30.413 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:30.413 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.2 2026-02-22T19:52:30.424 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:30.601 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.3] 2026-02-22T19:52:30.601 INFO:teuthology.orchestra.run.trial017.stdout: key = AQAzXptpQqukGBAALSufzKqx2N65ZkN1j7QxMw== 2026-02-22T19:52:30.601 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:30.601 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:30.601 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.3 2026-02-22T19:52:30.612 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:30.794 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror] 2026-02-22T19:52:30.794 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBYXptpBVryMBAAcB7Hj+R9oo7GjhU/BCXhLA== 2026-02-22T19:52:30.794 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror-peer" 2026-02-22T19:52:30.794 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:30.794 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror 2026-02-22T19:52:30.806 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:30.986 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.0] 2026-02-22T19:52:30.987 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBYXptpyiHzNBAA8+tcX3POTdJefVg/SESDVQ== 2026-02-22T19:52:30.987 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:30.987 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:30.987 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.0 2026-02-22T19:52:30.998 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:31.180 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.1] 2026-02-22T19:52:31.180 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBYXptpX7nFOBAAYRelmmdP5xhN9VFo7xRHsg== 2026-02-22T19:52:31.180 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:31.180 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:31.181 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.1 2026-02-22T19:52:31.191 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:31.367 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.2] 2026-02-22T19:52:31.367 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBZXptpbS73ABAA8HefkQ39p8T3vOCW+GrNGg== 2026-02-22T19:52:31.367 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:31.367 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:31.368 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.2 2026-02-22T19:52:31.379 DEBUG:teuthology.orchestra.run.trial017:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-22T19:52:31.559 INFO:teuthology.orchestra.run.trial017.stdout:[client.mirror.3] 2026-02-22T19:52:31.560 INFO:teuthology.orchestra.run.trial017.stdout: key = AQBZXptprCG7BBAAfqHj+Z+AWP8CbQgjCMqnIw== 2026-02-22T19:52:31.560 INFO:teuthology.orchestra.run.trial017.stdout: caps mon = "profile rbd-mirror" 2026-02-22T19:52:31.560 INFO:teuthology.orchestra.run.trial017.stdout: caps osd = "profile rbd" 2026-02-22T19:52:31.560 INFO:teuthology.orchestra.run.trial017.stderr:updated caps for client.mirror.3 2026-02-22T19:52:31.571 INFO:teuthology.run_tasks:Running task workunit... 2026-02-22T19:52:31.596 INFO:tasks.workunit:Pulling workunits from ref f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:52:31.597 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-22T19:52:31.597 DEBUG:teuthology.orchestra.run.trial017:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-22T19:52:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T19:52:31.602 INFO:teuthology.orchestra.run.trial017.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2026-02-22T19:52:31.602 DEBUG:teuthology.orchestra.run.trial017:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-22T19:52:31.646 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-22T19:52:31.646 DEBUG:teuthology.orchestra.run.trial017:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2026-02-22T19:52:31.690 INFO:tasks.workunit:timeout=3h 2026-02-22T19:52:31.690 INFO:tasks.workunit:cleanup=True 2026-02-22T19:52:31.691 DEBUG:teuthology.orchestra.run.trial017:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T19:52:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2026-02-22T19:52:53.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Note: switching to 'f90f4eaebb28b619cf76e80cbbe997656da69579'. 2026-02-22T19:52:53.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T19:52:53.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-22T19:52:53.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-22T19:52:53.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:state without impacting any branches by switching back to a branch. 2026-02-22T19:52:53.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: git switch -c 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Or undo this operation with: 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: git switch - 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T19:52:53.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:HEAD is now at f90f4eaebb2 Merge branch 'wip-74712-squid' of https://github.com/idryomov/ceph into wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T19:52:53.578 DEBUG:teuthology.orchestra.run.trial017:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2026-02-22T19:52:53.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-22T19:52:53.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2026-02-22T19:52:53.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-22T19:52:53.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-22T19:52:53.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-22T19:52:53.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2026-02-22T19:52:53.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2026-02-22T19:52:53.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-22T19:52:53.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2026-02-22T19:52:53.804 DEBUG:teuthology.orchestra.run.trial017:> set -ex 2026-02-22T19:52:53.804 DEBUG:teuthology.orchestra.run.trial017:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2026-02-22T19:52:53.850 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror.sh on cluster1.client.mirror... 2026-02-22T19:52:53.852 INFO:tasks.workunit:Running workunit rbd/rbd_mirror.sh... 2026-02-22T19:52:53.852 DEBUG:teuthology.orchestra.run.trial017:workunit test rbd/rbd_mirror.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror.sh 2026-02-22T19:52:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -gt 0 ']' 2026-02-22T19:52:53.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot '!=' snapshot ']' 2026-02-22T19:52:53.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror.sh 2026-02-22T19:52:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2026-02-22T19:52:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ type xmlstarlet 2026-02-22T19:52:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ XMLSTARLET=xmlstarlet 2026-02-22T19:52:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ RBD_MIRROR_INSTANCES=4 2026-02-22T19:52:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CLUSTER1=cluster1 2026-02-22T19:52:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CLUSTER2=cluster2 2026-02-22T19:52:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ PEER_CLUSTER_SUFFIX= 2026-02-22T19:52:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ POOL=mirror 2026-02-22T19:52:53.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ PARENT_POOL=mirror_parent 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ NS1=ns1 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ NS2=ns2 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ TEMPDIR= 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ID=mirror 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ RBD_MIRROR_MODE=snapshot 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ MIRROR_POOL_MODE=pool 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' snapshot = snapshot ']' 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ MIRROR_POOL_MODE=image 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ export 'CEPH_ARGS=--id mirror' 2026-02-22T19:52:53.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS='--id mirror' 2026-02-22T19:52:53.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ LAST_MIRROR_INSTANCE=3 2026-02-22T19:52:53.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror.sh 2026-02-22T19:52:53.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2026-02-22T19:52:53.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2026-02-22T19:52:53.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_BIN=. 2026-02-22T19:52:53.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_SRC=. 2026-02-22T19:52:53.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' -e CMakeCache.txt ']' 2026-02-22T19:52:53.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 -gt 0 ']' 2026-02-22T19:52:53.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ setup 2026-02-22T19:52:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local c 2026-02-22T19:52:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ trap 'cleanup $?' INT TERM EXIT 2026-02-22T19:52:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ setup_tempdir 2026-02-22T19:52:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:52:53.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mktemp -d 2026-02-22T19:52:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ TEMPDIR=/tmp/tmp.bhaDYZBKee 2026-02-22T19:52:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z 1 ']' 2026-02-22T19:52:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ setup_pools cluster1 cluster2 2026-02-22T19:52:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:52:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local remote_cluster=cluster2 2026-02-22T19:52:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mon_map_file 2026-02-22T19:52:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mon_addr 2026-02-22T19:52:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local admin_key_file 2026-02-22T19:52:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local uuid 2026-02-22T19:52:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:52:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2026-02-22T19:52:54.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:pool 'mirror' already exists 2026-02-22T19:52:54.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:52:54.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2026-02-22T19:52:55.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:pool 'mirror_parent' already exists 2026-02-22T19:52:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:52:55.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 pool init mirror 2026-02-22T19:52:58.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:52:58.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2026-02-22T19:53:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 1 ']' 2026-02-22T19:53:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ PEER_CLUSTER_SUFFIX=-DNE 2026-02-22T19:53:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1-DNE mirror image 2026-02-22T19:53:01.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2026-02-22T19:53:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2026-02-22T19:53:01.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2026-02-22T19:53:01.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 namespace create mirror_parent/ns1 2026-02-22T19:53:01.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2026-02-22T19:53:01.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2026-02-22T19:53:01.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent/ns1 image 2026-02-22T19:53:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z ']' 2026-02-22T19:53:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z 1 ']' 2026-02-22T19:53:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mon_map_file=/tmp/tmp.bhaDYZBKee/cluster2.monmap 2026-02-22T19:53:01.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:01.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster2 mon getmap 2026-02-22T19:53:02.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:got monmap epoch 1 2026-02-22T19:53:02.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ monmaptool --print /tmp/tmp.bhaDYZBKee/cluster2.monmap 2026-02-22T19:53:02.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -E 'mon\.' 2026-02-22T19:53:02.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ head -n 1 2026-02-22T19:53:02.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -E 's/^[0-9]+: ([^ ]+).+$/\1/' 2026-02-22T19:53:02.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -E 's/\/[0-9]+//g' 2026-02-22T19:53:02.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mon_addr='[v2:10.20.193.17:3300,v1:10.20.193.17:6789]' 2026-02-22T19:53:02.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_key_file=/tmp/tmp.bhaDYZBKee/cluster2.client.mirror.key 2026-02-22T19:53:02.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:02.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster2 auth get-key client.mirror 2026-02-22T19:53:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ peer_add cluster1 mirror client.mirror@cluster2-DNE '' --remote-mon-host '[v2:10.20.193.17:3300,v1:10.20.193.17:6789]' --remote-key-file /tmp/tmp.bhaDYZBKee/cluster2.client.mirror.key 2026-02-22T19:53:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local client_cluster=client.mirror@cluster2-DNE 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local remote_cluster=cluster2-DNE 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local uuid_var_name 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local error_code 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local peer_uuid 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 16 32 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:53:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror client.mirror@cluster2-DNE --remote-mon-host '[v2:10.20.193.17:3300,v1:10.20.193.17:6789]' --remote-key-file /tmp/tmp.bhaDYZBKee/cluster2.client.mirror.key 2026-02-22T19:53:02.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ peer_uuid=c42be08c-54e0-47b4-8faa-b0889455bae6 2026-02-22T19:53:02.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T19:53:02.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:53:02.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 17 ']' 2026-02-22T19:53:02.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 -eq 0 2026-02-22T19:53:02.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:02.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:02.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ peer_add cluster1 mirror_parent client.mirror@cluster2-DNE uuid 2026-02-22T19:53:02.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:02.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:02.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T19:53:02.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:02.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local client_cluster=client.mirror@cluster2-DNE 2026-02-22T19:53:02.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:02.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local remote_cluster=cluster2-DNE 2026-02-22T19:53:02.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local uuid_var_name 2026-02-22T19:53:02.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n uuid ']' 2026-02-22T19:53:02.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ uuid_var_name=uuid 2026-02-22T19:53:02.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:02.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local error_code 2026-02-22T19:53:02.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local peer_uuid 2026-02-22T19:53:02.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 16 32 2026-02-22T19:53:02.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:53:02.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent client.mirror@cluster2-DNE 2026-02-22T19:53:02.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ peer_uuid=4a3ca826-0c80-43e0-bfc9-77783b303376 2026-02-22T19:53:02.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T19:53:02.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:53:02.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 17 ']' 2026-02-22T19:53:02.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 -eq 0 2026-02-22T19:53:02.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n uuid ']' 2026-02-22T19:53:02.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval uuid=4a3ca826-0c80-43e0-bfc9-77783b303376 2026-02-22T19:53:02.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ uuid=4a3ca826-0c80-43e0-bfc9-77783b303376 2026-02-22T19:53:02.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:02.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:02.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror pool peer set mirror_parent 4a3ca826-0c80-43e0-bfc9-77783b303376 mon-host '[v2:10.20.193.17:3300,v1:10.20.193.17:6789]' 2026-02-22T19:53:02.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:02.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror pool peer set mirror_parent 4a3ca826-0c80-43e0-bfc9-77783b303376 key-file /tmp/tmp.bhaDYZBKee/cluster2.client.mirror.key 2026-02-22T19:53:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ setup_pools cluster2 cluster1 2026-02-22T19:53:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local remote_cluster=cluster1 2026-02-22T19:53:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mon_map_file 2026-02-22T19:53:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mon_addr 2026-02-22T19:53:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local admin_key_file 2026-02-22T19:53:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local uuid 2026-02-22T19:53:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:02.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2026-02-22T19:53:03.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:pool 'mirror' already exists 2026-02-22T19:53:03.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:03.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2026-02-22T19:53:04.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:pool 'mirror_parent' already exists 2026-02-22T19:53:04.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:04.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 pool init mirror 2026-02-22T19:53:07.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:07.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2026-02-22T19:53:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 1 ']' 2026-02-22T19:53:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ PEER_CLUSTER_SUFFIX=-DNE 2026-02-22T19:53:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2-DNE mirror image 2026-02-22T19:53:10.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2026-02-22T19:53:10.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2026-02-22T19:53:10.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2026-02-22T19:53:10.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 namespace create mirror_parent/ns1 2026-02-22T19:53:10.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2026-02-22T19:53:10.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2026-02-22T19:53:10.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent/ns1 image 2026-02-22T19:53:10.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z ']' 2026-02-22T19:53:10.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z 1 ']' 2026-02-22T19:53:10.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mon_map_file=/tmp/tmp.bhaDYZBKee/cluster1.monmap 2026-02-22T19:53:10.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:10.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster1 mon getmap 2026-02-22T19:53:10.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:got monmap epoch 1 2026-02-22T19:53:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ monmaptool --print /tmp/tmp.bhaDYZBKee/cluster1.monmap 2026-02-22T19:53:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -E 'mon\.' 2026-02-22T19:53:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ head -n 1 2026-02-22T19:53:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -E 's/^[0-9]+: ([^ ]+).+$/\1/' 2026-02-22T19:53:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -E 's/\/[0-9]+//g' 2026-02-22T19:53:10.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mon_addr='[v2:10.20.193.7:3300,v1:10.20.193.7:6789]' 2026-02-22T19:53:10.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_key_file=/tmp/tmp.bhaDYZBKee/cluster1.client.mirror.key 2026-02-22T19:53:10.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:10.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster1 auth get-key client.mirror 2026-02-22T19:53:10.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:10.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ peer_add cluster2 mirror client.mirror@cluster1-DNE '' --remote-mon-host '[v2:10.20.193.7:3300,v1:10.20.193.7:6789]' --remote-key-file /tmp/tmp.bhaDYZBKee/cluster1.client.mirror.key 2026-02-22T19:53:10.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:10.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:10.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:10.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:10.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local client_cluster=client.mirror@cluster1-DNE 2026-02-22T19:53:10.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:10.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local remote_cluster=cluster1-DNE 2026-02-22T19:53:10.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local uuid_var_name 2026-02-22T19:53:10.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:10.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local error_code 2026-02-22T19:53:10.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local peer_uuid 2026-02-22T19:53:10.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 16 32 2026-02-22T19:53:10.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:53:10.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror client.mirror@cluster1-DNE --remote-mon-host '[v2:10.20.193.7:3300,v1:10.20.193.7:6789]' --remote-key-file /tmp/tmp.bhaDYZBKee/cluster1.client.mirror.key 2026-02-22T19:53:10.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ peer_uuid=e056ba3a-4324-4790-8cdf-40b7035d9459 2026-02-22T19:53:10.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T19:53:10.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:53:10.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 17 ']' 2026-02-22T19:53:10.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 -eq 0 2026-02-22T19:53:10.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:10.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:10.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ peer_add cluster2 mirror_parent client.mirror@cluster1-DNE uuid 2026-02-22T19:53:10.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:10.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:10.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T19:53:10.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:10.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local client_cluster=client.mirror@cluster1-DNE 2026-02-22T19:53:10.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:10.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local remote_cluster=cluster1-DNE 2026-02-22T19:53:10.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local uuid_var_name 2026-02-22T19:53:10.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n uuid ']' 2026-02-22T19:53:10.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ uuid_var_name=uuid 2026-02-22T19:53:10.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:10.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local error_code 2026-02-22T19:53:10.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local peer_uuid 2026-02-22T19:53:10.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 16 32 2026-02-22T19:53:10.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:53:10.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent client.mirror@cluster1-DNE 2026-02-22T19:53:10.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ peer_uuid=ec513268-fe11-4664-a2da-d22ebb2bda08 2026-02-22T19:53:10.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T19:53:10.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:53:10.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 17 ']' 2026-02-22T19:53:10.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 -eq 0 2026-02-22T19:53:10.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n uuid ']' 2026-02-22T19:53:10.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval uuid=ec513268-fe11-4664-a2da-d22ebb2bda08 2026-02-22T19:53:10.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ uuid=ec513268-fe11-4664-a2da-d22ebb2bda08 2026-02-22T19:53:10.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:10.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:10.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror pool peer set mirror_parent ec513268-fe11-4664-a2da-d22ebb2bda08 mon-host '[v2:10.20.193.7:3300,v1:10.20.193.7:6789]' 2026-02-22T19:53:10.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T19:53:10.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror pool peer set mirror_parent ec513268-fe11-4664-a2da-d22ebb2bda08 key-file /tmp/tmp.bhaDYZBKee/cluster1.client.mirror.key 2026-02-22T19:53:10.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:10.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: add image and test replay' 2026-02-22T19:53:10.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T19:53:10.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T19:53:10.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 19:53:10 TEST: add image and test replay 2026-02-22T19:53:10.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 19:53:10 TEST: add image and test replay 2026-02-22T19:53:10.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirrors cluster1 2026-02-22T19:53:10.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:10.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:53:10.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:10.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:0 2026-02-22T19:53:10.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:53:10.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:10.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:10.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:1 2026-02-22T19:53:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:53:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:53:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:53:10.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:10.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:10.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:10.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:53:10.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:10.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:10.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:2 2026-02-22T19:53:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:53:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:53:10.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:53:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:53:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:10.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:11.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:11.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:3 2026-02-22T19:53:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:53:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:53:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:53:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:11.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:11.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:11.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:53:11.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:11.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:11.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:11.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image=test 2026-02-22T19:53:11.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 2026-02-22T19:53:11.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:11.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:11.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:11.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:11.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:11.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:11.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T19:53:11.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T19:53:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test 2026-02-22T19:53:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T19:53:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:11.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test 2026-02-22T19:53:11.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T19:53:11.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:11.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T19:53:11.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:11.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test snapshot 2026-02-22T19:53:11.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:11.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:11.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:11.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T19:53:11.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2026-02-22T19:53:12.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T19:53:12.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test 2026-02-22T19:53:12.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test': 2026-02-22T19:53:12.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T19:53:12.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T19:53:12.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T19:53:12.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 10bb7145ac75 2026-02-22T19:53:12.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.10bb7145ac75 2026-02-22T19:53:12.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T19:53:12.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T19:53:12.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T19:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T19:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 19:53:11 2026 2026-02-22T19:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 19:53:11 2026 2026-02-22T19:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 19:53:11 2026 2026-02-22T19:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 10bb7145ac75 2026-02-22T19:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T19:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T19:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T19:53:12.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_image_meta cluster2 mirror test key1 value1 2026-02-22T19:53:12.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:12.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:12.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:12.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local key=key1 2026-02-22T19:53:12.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=value1 2026-02-22T19:53:12.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key1 value1 2026-02-22T19:53:12.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_image_meta cluster2 mirror test key2 value2 2026-02-22T19:53:12.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:12.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:12.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:12.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local key=key2 2026-02-22T19:53:12.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=value2 2026-02-22T19:53:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key2 value2 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:12.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:13.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:13.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:13.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:13.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:13.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:13.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:13.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:13.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:13.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:13.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:13.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:13.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:13.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:13.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:13.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:13.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:13.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:13.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:13.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:13.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:13.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:13.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:53:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:13.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:13.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:13.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:13.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:13.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:13.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:13.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:13.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:13.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:13.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T19:53:13.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T19:53:13.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:13.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:13.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T19:53:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:13.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T19:53:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T19:53:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:53:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:13.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T19:53:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:13.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:13.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:13.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:13.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T19:53:13.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:53:13.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:13.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:13.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T19:53:14.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:14.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:14.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:14.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:14.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:14.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:14.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:14.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:14.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:14.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:14.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:14.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:14.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:14.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:14.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:14.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:53:14.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:53:14.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:14.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:53:14.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:53:14.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:53:14.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test 100 2026-02-22T19:53:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T19:53:14.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T19:53:14.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:14.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T19:53:14.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T19:53:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T19:53:14.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T19:53:14.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.17 bytes/sec: 24 MiB/s 2026-02-22T19:53:14.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:53:14.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:53:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:53:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:53:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:53:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:53:15.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 4 2026-02-22T19:53:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:53:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:53:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:15.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:15.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:53:15.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:53:15.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:15.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:53:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:53:15.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:53:15.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:15.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:53:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:53:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:53:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:53:15.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:53:15.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:53:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=4 2026-02-22T19:53:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:53:15.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:53:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:53:15.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:53:15.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:15.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:15.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:15.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:53:15.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:53:15.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:53:15.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:53:15.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=3 2026-02-22T19:53:15.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 4 = 3 2026-02-22T19:53:15.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:53:15.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T19:53:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:53:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:15.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:53:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:53:15.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:53:15.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=4 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 4 = 4 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:15.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:53:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:16.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:16.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:53:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771789992,"replay_state":"idle"} 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:14' 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:53:16.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:53:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771789992,"replay_state":"idle"} 2026-02-22T19:53:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:16.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:53:16.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:16.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:16.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:14' 2026-02-22T19:53:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:53:14' 2026-02-22T19:53:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:14' 2026-02-22T19:53:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:16.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING142804287mirror.0trial01719.2.3-689-gf90f4eaetrueOK42844293mirror.1trial01719.2.3-689-gf90f4eaefalseOK42914300mirror.2trial01719.2.3-689-gf90f4eaefalseOK42974303mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771789992,"replay_state":"idle"}42804287mirror.0trial0172026-02-22 19:53:14' 2026-02-22T19:53:16.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:16.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:53:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:14' 2026-02-22T19:53:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:53:16.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:53:16.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:16.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:53:16.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z '' ']' 2026-02-22T19:53:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test down+unknown 2026-02-22T19:53:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:16.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=down+unknown 2026-02-22T19:53:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:17.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test down+unknown '' '' 2026-02-22T19:53:17.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:17.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:17.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=down+unknown 2026-02-22T19:53:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:53:17.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:53:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:53:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T19:53:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T19:53:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22T19:53:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:53:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:53:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:53:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771789992,"replay_state":"idle"} 2026-02-22T19:53:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:14 2026-02-22T19:53:17.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:53:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 3 .mirror.primary.d3facf73-8492-44d6-bde1-34803b6b4fd6.9a549407-ef62-4a0e-9d84-8c3660deaa55 (peer_uuids:[]) 2026-02-22T19:53:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 4 .mirror.primary.d3facf73-8492-44d6-bde1-34803b6b4fd6.dfeaa274-6562-404e-aa75-797d5beb2e07 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:53:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*down+unknown' 2026-02-22T19:53:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: down+unknown 2026-02-22T19:53:17.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:17.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: status not found 2026-02-22T19:53:17.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:17.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo down+unknown 2026-02-22T19:53:17.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:17.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: ' 2026-02-22T19:53:17.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:17.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:17.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update= 2026-02-22T19:53:17.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test down+unknown '' 2026-02-22T19:53:17.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:17.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:17.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:17.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=down+unknown 2026-02-22T19:53:17.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local prev_last_update= 2026-02-22T19:53:17.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:17.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:17.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:17.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='UNKNOWNUNKNOWNOK1testd3facf73-8492-44d6-bde1-34803b6b4fd6down+unknownstatus not foundcluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771789992,"replay_state":"idle"}2026-02-22 19:53:14' 2026-02-22T19:53:17.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:17.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:53:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update= 2026-02-22T19:53:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:53:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=down+unknown 2026-02-22T19:53:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo down+unknown 2026-02-22T19:53:17.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep down+unknown 2026-02-22T19:53:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:down+unknown 2026-02-22T19:53:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test 2026-02-22T19:53:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T19:53:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.export 2026-02-22T19:53:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.export 2026-02-22T19:53:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:17.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:53:17.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.export 2026-02-22T19:53:17.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.export 2026-02-22T19:53:17.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:17.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:53:17.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:53:17.995 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:53:18.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:53:18.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:53:18.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:53:18.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:53:18.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:53:18.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_image_meta cluster1 mirror test key1 value1 2026-02-22T19:53:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local key=key1 2026-02-22T19:53:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local value=value1 2026-02-22T19:53:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 image-meta get mirror/test key1 2026-02-22T19:53:18.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test value1 = value1 2026-02-22T19:53:18.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_image_meta cluster1 mirror test key2 value2 2026-02-22T19:53:18.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:18.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:18.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:18.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local key=key2 2026-02-22T19:53:18.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local value=value2 2026-02-22T19:53:18.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 image-meta get mirror/test key2 2026-02-22T19:53:18.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test value2 = value2 2026-02-22T19:53:18.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: stop mirror, add image, start mirror and test replay' 2026-02-22T19:53:18.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T19:53:18.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T19:53:18.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 19:53:18 TEST: stop mirror, add image, start mirror and test replay 2026-02-22T19:53:18.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 19:53:18 TEST: stop mirror, add image, start mirror and test replay 2026-02-22T19:53:18.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirrors cluster1 2026-02-22T19:53:18.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:18.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T19:53:18.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:53:18.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:18.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:0 '' 2026-02-22T19:53:18.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:53:18.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T19:53:18.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:18.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T19:53:18.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:0 2026-02-22T19:53:18.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:0 2026-02-22T19:53:18.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:18.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T19:53:18.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T19:53:18.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T19:53:18.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=20740 2026-02-22T19:53:18.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 20740 ']' 2026-02-22T19:53:18.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 20740 2026-02-22T19:53:18.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T19:53:18.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:53:18.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:18.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20740 '$2 == pid {print; exit 1}' 2026-02-22T19:53:18.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 20740 1.7 0.0 568184 41492 ? Ssl 19:53 0:00 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:18.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T19:53:18.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:19.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:19.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20740 '$2 == pid {print; exit 1}' 2026-02-22T19:53:19.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T19:53:19.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:19.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20740 '$2 == pid {print; exit 1}' 2026-02-22T19:53:19.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:19.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:53:19.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:19.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:19.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:19.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:19.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:19.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:19.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:19.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:19.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:19.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:19.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:19.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:19.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:19.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:19.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:19.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:19.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-22T19:53:19.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:53:19.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:53:19.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:19.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:19.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:19.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:19.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:19.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:19.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:19.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:19.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:19.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:19.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:19.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:19.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:19.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:19.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:0 2026-02-22T19:53:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:19.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:19.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:19.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:19.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:19.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:19.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T19:53:19.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T19:53:19.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T19:53:19.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:19.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:1 '' 2026-02-22T19:53:19.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:53:19.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T19:53:19.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:19.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T19:53:19.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:1 2026-02-22T19:53:19.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:1 2026-02-22T19:53:19.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:19.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:19.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:53:19.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T19:53:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T19:53:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T19:53:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:53:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:53:19.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T19:53:19.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T19:53:19.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T19:53:19.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=20768 2026-02-22T19:53:19.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 20768 ']' 2026-02-22T19:53:19.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 20768 2026-02-22T19:53:19.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T19:53:19.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:53:19.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:19.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20768 '$2 == pid {print; exit 1}' 2026-02-22T19:53:19.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 20768 0.7 0.0 701368 35444 ? Ssl 19:53 0:00 rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:19.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T19:53:19.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:20.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20768 '$2 == pid {print; exit 1}' 2026-02-22T19:53:20.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T19:53:20.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:20.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20768 '$2 == pid {print; exit 1}' 2026-02-22T19:53:20.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:20.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:53:20.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:20.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:20.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:20.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:20.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:20.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:20.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:20.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:20.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:20.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:20.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:20.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:20.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:20.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:20.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-22T19:53:20.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:53:20.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:53:20.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:20.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:20.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:20.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:20.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:20.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:1 2026-02-22T19:53:20.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:20.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:20.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:20.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:20.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:20.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:20.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:20.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:20.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:20.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:20.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:20.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:20.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:20.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T19:53:20.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T19:53:20.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T19:53:20.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:20.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:2 '' 2026-02-22T19:53:20.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:53:20.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T19:53:20.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:20.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:2 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:2 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T19:53:20.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T19:53:20.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T19:53:20.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T19:53:20.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T19:53:20.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T19:53:20.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T19:53:20.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=20828 2026-02-22T19:53:20.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 20828 ']' 2026-02-22T19:53:20.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 20828 2026-02-22T19:53:20.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T19:53:20.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:53:20.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:20.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20828 '$2 == pid {print; exit 1}' 2026-02-22T19:53:20.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 20828 0.7 0.0 611216 35592 ? Rsl 19:53 0:00 rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:20.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T19:53:20.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:21.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20828 '$2 == pid {print; exit 1}' 2026-02-22T19:53:21.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T19:53:21.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:21.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20828 '$2 == pid {print; exit 1}' 2026-02-22T19:53:21.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:21.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:53:21.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:21.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:21.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:21.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:53:21.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:21.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:21.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:21.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:21.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:21.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:21.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:21.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:53:21.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:21.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:21.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:21.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:21.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-22T19:53:21.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:53:21.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:21.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:21.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:21.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:21.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:2 2026-02-22T19:53:21.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T19:53:21.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:21.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:21.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:53:21.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:21.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:21.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:21.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:21.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:21.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:21.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:21.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:53:21.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:21.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T19:53:21.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T19:53:21.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T19:53:21.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:21.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:3 '' 2026-02-22T19:53:21.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:53:21.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T19:53:21.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:21.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T19:53:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:3 2026-02-22T19:53:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:3 2026-02-22T19:53:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T19:53:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T19:53:21.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:21.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:21.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:21.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:21.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T19:53:21.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T19:53:21.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T19:53:21.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T19:53:21.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T19:53:21.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T19:53:21.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T19:53:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=20873 2026-02-22T19:53:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 20873 ']' 2026-02-22T19:53:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 20873 2026-02-22T19:53:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T19:53:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:53:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:21.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20873 '$2 == pid {print; exit 1}' 2026-02-22T19:53:21.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 20873 0.7 0.0 701372 35868 ? Ssl 19:53 0:00 rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:21.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T19:53:21.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:22.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:22.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20873 '$2 == pid {print; exit 1}' 2026-02-22T19:53:22.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T19:53:22.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T19:53:22.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=20873 '$2 == pid {print; exit 1}' 2026-02-22T19:53:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:53:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:53:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:53:22.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:22.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:22.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:22.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:22.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-22T19:53:22.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:53:22.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:53:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:22.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:22.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:22.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:3 2026-02-22T19:53:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T19:53:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:53:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:22.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:22.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:22.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:53:22.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:22.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T19:53:22.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T19:53:22.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T19:53:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image1=test1 2026-02-22T19:53:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test1 snapshot 2026-02-22T19:53:22.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T19:53:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T19:53:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test1 2026-02-22T19:53:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:22.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:22.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:22.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:22.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:22.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:22.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T19:53:22.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:22.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test1 2026-02-22T19:53:22.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T19:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T19:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test1 snapshot 2026-02-22T19:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T19:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test1 snapshot 2026-02-22T19:53:23.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T19:53:23.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test1 2026-02-22T19:53:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test1': 2026-02-22T19:53:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T19:53:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T19:53:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T19:53:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 10e79c7fc3f2 2026-02-22T19:53:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.10e79c7fc3f2 2026-02-22T19:53:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T19:53:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T19:53:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T19:53:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T19:53:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 19:53:22 2026 2026-02-22T19:53:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 19:53:22 2026 2026-02-22T19:53:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 19:53:22 2026 2026-02-22T19:53:23.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 10e79c7fc3f2 2026-02-22T19:53:23.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T19:53:23.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T19:53:23.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:53:23.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T19:53:23.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test1 100 2026-02-22T19:53:23.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:23.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:23.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T19:53:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T19:53:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T19:53:23.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test1 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T19:53:23.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T19:53:23.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T19:53:23.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.15 bytes/sec: 24 MiB/s 2026-02-22T19:53:23.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirrors cluster1 2026-02-22T19:53:23.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:23.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:53:23.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:23.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:0 2026-02-22T19:53:23.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:53:23.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:23.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:23.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:53:23.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:23.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:23.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:23.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:53:23.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:23.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:23.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:23.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:53:23.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:23.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:23.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:23.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:23.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:1 2026-02-22T19:53:23.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:53:23.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:23.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:23.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:53:23.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:23.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:23.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:23.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:53:23.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:23.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:23.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:23.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:53:23.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:23.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:23.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:23.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:23.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:2 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:23.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:53:23.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:23.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:23.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:23.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:23.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:3 2026-02-22T19:53:23.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:53:23.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:23.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:23.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:53:23.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:23.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:23.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:23.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:53:23.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:23.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:23.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:23.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:53:23.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:23.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:23.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:23.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:24.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-22T19:53:24.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:24.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:24.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:24.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:53:24.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:24.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:24.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-22T19:53:24.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:24.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:24.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:24.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:24.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:24.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:24.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:24.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:24.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:24.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:24.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:24.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:53:24.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:24.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:24.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:24.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:24.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:24.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:24.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:24.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:24.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:24.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:24.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:53:24.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-22T19:53:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:24.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test1 2026-02-22T19:53:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:24.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:24.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:24.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:24.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:24.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:24.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:24.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:24.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:24.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:53:24.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:53:24.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:24.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:24.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:24.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:24.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test1 2026-02-22T19:53:24.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-22T19:53:24.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:24.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test1 2026-02-22T19:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T19:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T19:53:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T19:53:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T19:53:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:24.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test1 2026-02-22T19:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-22T19:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test1 2026-02-22T19:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T19:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T19:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T19:53:24.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T19:53:24.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:24.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:24.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:24.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:24.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test1 2026-02-22T19:53:24.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test1'] 2026-02-22T19:53:24.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:53:24.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:24.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:24.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T19:53:25.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:25.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:25.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:25.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:25.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:25.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:25.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:25.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:25.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:25.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:25.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:25.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:25.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:25.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:25.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:25.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:25.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:25.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:53:25.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:25.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:53:25.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:53:25.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:25.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test1 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test1 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:25.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test1 2026-02-22T19:53:26.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 6 2026-02-22T19:53:26.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test1 2026-02-22T19:53:26.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:53:26.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:26.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:26.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:26.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test1.status 2026-02-22T19:53:26.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test1.status 2026-02-22T19:53:26.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:26.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test1.status 2026-02-22T19:53:26.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test1.status 2026-02-22T19:53:26.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test1.status 2026-02-22T19:53:26.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:26.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test1.status 2026-02-22T19:53:26.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test1 /tmp/tmp.bhaDYZBKee/cluster2-mirror-test1.status 2026-02-22T19:53:26.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:26.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:26.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:26.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test1.status 2026-02-22T19:53:26.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test1 --format xml 2026-02-22T19:53:26.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:53:26.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:53:26.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=6 2026-02-22T19:53:26.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:53:26.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:53:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:53:26.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test1 /tmp/tmp.bhaDYZBKee/cluster1-mirror-test1.status 2026-02-22T19:53:26.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:26.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:26.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:26.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test1.status 2026-02-22T19:53:26.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test1 --format xml 2026-02-22T19:53:26.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:53:26.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:53:26.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=5 2026-02-22T19:53:26.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 6 = 5 2026-02-22T19:53:26.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:53:26.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T19:53:26.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test1 /tmp/tmp.bhaDYZBKee/cluster1-mirror-test1.status 2026-02-22T19:53:26.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:26.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:26.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:26.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test1.status 2026-02-22T19:53:26.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test1 --format xml 2026-02-22T19:53:26.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:53:26.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:53:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=6 2026-02-22T19:53:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 6 = 6 2026-02-22T19:53:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:26.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test1 2026-02-22T19:53:26.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:26.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:26.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:26.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:53:26.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2026-02-22T19:53:26.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:26.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:26.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:26.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:26.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:26.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:26.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:26.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2026-02-22T19:53:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:27.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:27.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-22T19:53:28.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test1: 2026-02-22T19:53:28.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:53:28.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:53:28.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790003,"replay_state":"idle"} 2026-02-22T19:53:28.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:53:28.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:25' 2026-02-22T19:53:28.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:53:28.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:53:28.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:28.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790003,"replay_state":"idle"} 2026-02-22T19:53:28.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:28.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:28.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:28.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:28.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:28.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:53:28.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:28.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:28.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:25' 2026-02-22T19:53:28.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-22 19:53:25' 2026-02-22T19:53:28.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:28.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:28.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:25' 2026-02-22T19:53:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:28.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGWARNINGWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK4355mirror.3trial01719.2.3-689-gf90f4eaefalseWARNINGnot reporting statustestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771789995,"remote_snapshot_timestamp":1771789995,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:53:27test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790003,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:53:25' 2026-02-22T19:53:28.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:28.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-22T19:53:28.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:25' 2026-02-22T19:53:28.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-22T19:53:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:53:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:53:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:28.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:28.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:28.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z '' ']' 2026-02-22T19:53:28.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test1 down+unknown 2026-02-22T19:53:28.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:28.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:28.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:28.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=down+unknown 2026-02-22T19:53:28.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:28.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:28.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:28.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test1 down+unknown '' '' 2026-02-22T19:53:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=down+unknown 2026-02-22T19:53:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test1 2026-02-22T19:53:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test1: 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790003,"replay_state":"idle"} 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:25 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 5 .mirror.primary.f4ee5839-bc21-4b89-840a-30463c6a4063.6d06465c-b67e-4b75-911a-2bb9eee62524 (peer_uuids:[]) 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 6 .mirror.primary.f4ee5839-bc21-4b89-840a-30463c6a4063.6f5a0fd2-473a-41cb-b7dd-3344662b00f3 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*down+unknown' 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: down+unknown 2026-02-22T19:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: status not found 2026-02-22T19:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo down+unknown 2026-02-22T19:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:29.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: ' 2026-02-22T19:53:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:29.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:29.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update= 2026-02-22T19:53:29.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test1 down+unknown '' 2026-02-22T19:53:29.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:29.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:29.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:29.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=down+unknown 2026-02-22T19:53:29.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local prev_last_update= 2026-02-22T19:53:29.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:29.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:29.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='UNKNOWNUNKNOWNOK2testd3facf73-8492-44d6-bde1-34803b6b4fd6down+unknownstatus not foundcluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771789995,"remote_snapshot_timestamp":1771789995,"replay_state":"idle"}2026-02-22 19:53:27test1f4ee5839-bc21-4b89-840a-30463c6a4063down+unknownstatus not foundcluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790003,"replay_state":"idle"}2026-02-22 19:53:25' 2026-02-22T19:53:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:29.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-22T19:53:29.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update= 2026-02-22T19:53:29.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-22T19:53:29.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=down+unknown 2026-02-22T19:53:29.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo down+unknown 2026-02-22T19:53:29.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep down+unknown 2026-02-22T19:53:29.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:down+unknown 2026-02-22T19:53:29.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:29.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:29.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test1 2026-02-22T19:53:29.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:29.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:29.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T19:53:29.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test1.export 2026-02-22T19:53:29.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test1.export 2026-02-22T19:53:29.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:29.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test1.export 2026-02-22T19:53:29.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test1.export 2026-02-22T19:53:29.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test1.export 2026-02-22T19:53:29.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:29.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test1.export 2026-02-22T19:53:29.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test1.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test1.export 2026-02-22T19:53:29.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test1 /tmp/tmp.bhaDYZBKee/cluster2-mirror-test1.export 2026-02-22T19:53:29.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:53:29.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test1 /tmp/tmp.bhaDYZBKee/cluster1-mirror-test1.export 2026-02-22T19:53:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test1.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test1.export 2026-02-22T19:53:29.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test1.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test1.export 2026-02-22T19:53:29.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:29.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: test the first image is replaying after restart' 2026-02-22T19:53:29.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T19:53:29.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T19:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 19:53:29 TEST: test the first image is replaying after restart 2026-02-22T19:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 19:53:29 TEST: test the first image is replaying after restart 2026-02-22T19:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test 100 2026-02-22T19:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T19:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T19:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T19:53:29.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T19:53:29.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T19:53:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T19:53:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.15 bytes/sec: 24 MiB/s 2026-02-22T19:53:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:53:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:53:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:53:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:30.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:53:30.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:30.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:30.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:30.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:53:30.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:30.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:30.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:53:30.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:30.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:30.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:30.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:30.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:30.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:30.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:30.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:30.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:30.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:30.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:30.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:30.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:31.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:53:31.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:53:31.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:31.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:31.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:53:31.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:31.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:31.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:31.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:31.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:53:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:53:31.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:31.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:53:31.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:53:31.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:53:31.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:31.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:53:31.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:53:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:53:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:53:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:53:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:53:33.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 7 2026-02-22T19:53:33.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:53:33.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:53:33.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:33.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:33.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:33.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:53:33.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:53:33.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:33.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:53:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:53:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:53:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:53:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:53:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:53:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:53:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:53:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:53:33.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:53:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=7 2026-02-22T19:53:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:53:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:53:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:53:33.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:53:33.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:33.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:33.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:33.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:53:33.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:53:33.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:53:33.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:53:33.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=7 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 7 = 7 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:34.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:53:34.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:34.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:34.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:34.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:34.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:34.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:34.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:34.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:34.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:53:34.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:53:34.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:53:34.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:53:34.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771789995,"remote_snapshot_timestamp":1771789995,"replay_state":"idle"} 2026-02-22T19:53:34.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:53:34.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:27' 2026-02-22T19:53:34.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:53:34.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:53:34.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:34.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771789995,"remote_snapshot_timestamp":1771789995,"replay_state":"idle"} 2026-02-22T19:53:34.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:34.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:34.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:34.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:34.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:34.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:53:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:34.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:27' 2026-02-22T19:53:34.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:53:27' 2026-02-22T19:53:34.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:34.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:27' 2026-02-22T19:53:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:34.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:34.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771789995,"remote_snapshot_timestamp":1771789995,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:53:27test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790003,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:53:25' 2026-02-22T19:53:34.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:34.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:53:34.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:27' 2026-02-22T19:53:34.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:53:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:53:34.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:34.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:53:34.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:34.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:34.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:34.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test 2026-02-22T19:53:34.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:34.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:34.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T19:53:34.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.export 2026-02-22T19:53:34.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.export 2026-02-22T19:53:34.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:34.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:53:34.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.export 2026-02-22T19:53:34.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.export 2026-02-22T19:53:34.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:53:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:53:34.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:53:34.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:53:34.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:53:34.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:53:34.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:53:34.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:53:34.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:53:34.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z '' ']' 2026-02-22T19:53:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: stop/start/restart mirror via admin socket' 2026-02-22T19:53:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T19:53:34.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T19:53:34.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 19:53:34 TEST: stop/start/restart mirror via admin socket 2026-02-22T19:53:34.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 19:53:34 TEST: stop/start/restart mirror via admin socket 2026-02-22T19:53:34.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ all_admin_daemons cluster1 rbd mirror stop 2026-02-22T19:53:34.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:34.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:34.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror stop 2026-02-22T19:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:34.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:53:34.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:34.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:34.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:34.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:34.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:34.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:34.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:34.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop 2026-02-22T19:53:34.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:34.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror stop 2026-02-22T19:53:34.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:53:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:53:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:53:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:34.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:34.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:34.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:34.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:34.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:34.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:34.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:34.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:34.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:34.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:34.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop 2026-02-22T19:53:34.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:2 rbd mirror stop 2026-02-22T19:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:34.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:53:34.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:34.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:34.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:34.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:34.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop 2026-02-22T19:53:35.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:35.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:3 rbd mirror stop 2026-02-22T19:53:35.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:53:35.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:35.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:35.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:35.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:53:35.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:35.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:35.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:53:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:53:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:35.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:35.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:35.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:35.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:35.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop 2026-02-22T19:53:35.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:53:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:53:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:53:35.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:35.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:35.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:36.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:53:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:53:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:53:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:36.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:36.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:36.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:36.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:36.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:36.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:53:36.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:53:36.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:36.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:36.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:36.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:36.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:36.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:36.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:36.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:36.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:36.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:36.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:36.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:36.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:36.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:36.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:36.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:36.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:36.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:36.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:53:36.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:53:36.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:36.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:36.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:36.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:36.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:36.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:36.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:53:36.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:53:36.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:53:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2026-02-22T19:53:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2026-02-22T19:53:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:53:36.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:36.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:36.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:37.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2026-02-22T19:53:37.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:37.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:37.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:37.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:53:37.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:37.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:37.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-22T19:53:37.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:37.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:37.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:37.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:37.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:37.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:37.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:37.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:37.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:37.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:37.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:37.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:53:37.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:37.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:37.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:37.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:37.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:37.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:37.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:37.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:37.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:37.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:53:37.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:53:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:53:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:53:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:53:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:37.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:37.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:37.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:37.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:38.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:53:38.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:38.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:38.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:38.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:38.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:38.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:38.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:38.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:38.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:53:38.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:53:38.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:53:38.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:53:38.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: stopped 2026-02-22T19:53:38.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:53:38.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:34' 2026-02-22T19:53:38.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:53:38.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:53:38.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:38.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: stopped 2026-02-22T19:53:38.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:38.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:53:38.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:38.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:53:38.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:38.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:53:38.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:38.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:34' 2026-02-22T19:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:53:34' 2026-02-22T19:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:34' 2026-02-22T19:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:38.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+stoppedstopped43424353mirror.1trial0172026-02-22 19:53:34test1f4ee5839-bc21-4b89-840a-30463c6a4063up+stoppedstopped43384346mirror.0trial0172026-02-22 19:53:34' 2026-02-22T19:53:38.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:38.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:53:38.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:34' 2026-02-22T19:53:38.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:53:38.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:53:38.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:53:38.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:53:38.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:53:38.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:38.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:38.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2026-02-22T19:53:38.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:38.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:38.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:38.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:38.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:38.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:38.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:38.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:39.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2026-02-22T19:53:39.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:39.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:39.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:39.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:39.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:39.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:39.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:39.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:39.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-22T19:53:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test1: 2026-02-22T19:53:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:53:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:53:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: stopped 2026-02-22T19:53:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:53:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:34' 2026-02-22T19:53:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:53:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:53:39.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:39.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: stopped 2026-02-22T19:53:39.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:39.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:53:39.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:39.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:53:39.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:39.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:53:39.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:39.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:39.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:34' 2026-02-22T19:53:39.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2026-02-22 19:53:34' 2026-02-22T19:53:39.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:39.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:39.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:39.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:34' 2026-02-22T19:53:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:39.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:39.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+stoppedstopped43424353mirror.1trial0172026-02-22 19:53:34test1f4ee5839-bc21-4b89-840a-30463c6a4063up+stoppedstopped43384346mirror.0trial0172026-02-22 19:53:34' 2026-02-22T19:53:39.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:39.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-22T19:53:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:34' 2026-02-22T19:53:39.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-22T19:53:39.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:53:39.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:53:39.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:53:39.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:53:39.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:39.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:39.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ all_admin_daemons cluster1 rbd mirror start 2026-02-22T19:53:39.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:39.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:39.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:53:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror start 2026-02-22T19:53:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:53:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:39.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:39.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:39.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:39.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:39.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:39.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:39.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror start 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:53:39.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:39.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:39.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:39.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:39.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start 2026-02-22T19:53:39.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:39.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:2 rbd mirror start 2026-02-22T19:53:39.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:53:39.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:39.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:39.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:39.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:53:39.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:39.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:39.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:39.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:53:39.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:39.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:39.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:39.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:39.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:39.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:39.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:53:39.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:39.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:39.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror start 2026-02-22T19:53:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:3 rbd mirror start 2026-02-22T19:53:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:53:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:39.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:39.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:53:39.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:39.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:39.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:39.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:39.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:39.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror start 2026-02-22T19:53:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:53:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:53:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:39.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:53:39.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:39.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:39.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:40.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:53:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:53:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:40.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:53:40.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:40.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:40.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:40.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:40.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:40.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:40.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:40.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:40.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:40.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:40.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:40.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:40.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:40.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:40.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:53:40.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:53:40.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:40.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:40.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:40.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:40.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:40.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:40.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:40.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:40.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:40.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:40.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:40.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:40.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:40.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:40.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:40.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:40.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:40.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:40.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:40.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:40.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:40.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:40.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:53:40.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:40.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:40.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:40.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:40.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:40.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:40.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:40.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:53:40.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:53:40.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:40.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:40.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:40.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:40.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:40.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:40.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:53:40.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:53:40.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:40.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:53:40.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:53:40.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:53:40.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-22T19:53:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-22T19:53:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:53:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:40.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:41.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-22T19:53:41.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:41.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:41.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:41.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:53:41.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:41.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-22T19:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:41.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:41.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:41.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:41.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:41.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:41.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:41.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:53:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:41.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:41.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:41.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:41.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:53:42.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:42.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:53:42.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:53:42.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:42.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:53:42.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:53:42.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:53:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:53:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:43.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:53:43.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:43.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:43.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:43.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:43.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:43.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:43.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:43.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:43.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:53:43.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:53:43.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:53:43.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:53:43.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:53:43.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:53:43.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:39' 2026-02-22T19:53:43.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:53:43.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:53:43.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:43.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:53:43.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:43.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:43.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:43.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:43.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:43.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:53:43.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:43.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:43.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:39' 2026-02-22T19:53:43.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:53:39' 2026-02-22T19:53:43.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:43.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:43.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:43.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:43.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:39' 2026-02-22T19:53:43.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:43.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:43.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:43.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:53:39test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:53:39' 2026-02-22T19:53:43.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:43.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:53:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:39' 2026-02-22T19:53:43.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:53:43.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:53:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:43.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:53:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2026-02-22T19:53:43.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:43.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2026-02-22T19:53:44.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:44.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:44.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:44.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:44.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:44.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:44.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:44.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:44.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-22T19:53:44.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test1: 2026-02-22T19:53:44.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:53:44.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:53:44.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"} 2026-02-22T19:53:44.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:53:44.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:39' 2026-02-22T19:53:44.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:53:44.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:53:44.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:44.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"} 2026-02-22T19:53:44.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:44.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:44.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:53:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:39' 2026-02-22T19:53:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-22 19:53:39' 2026-02-22T19:53:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:44.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:39' 2026-02-22T19:53:44.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:44.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:44.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:44.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:53:39test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:53:39' 2026-02-22T19:53:44.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:44.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-22T19:53:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:39' 2026-02-22T19:53:44.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-22T19:53:44.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:53:44.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:44.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:53:44.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:44.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:44.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:44.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ all_admin_daemons cluster1 rbd mirror restart 2026-02-22T19:53:44.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:44.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:44.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:53:44.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:44.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror restart 2026-02-22T19:53:44.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:53:44.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:44.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:44.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:44.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:53:44.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:44.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:44.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:44.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:44.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:44.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:44.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:44.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:44.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:44.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:44.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:44.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:44.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:44.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror restart 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:44.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:44.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:44.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:44.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:44.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:44.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:2 rbd mirror restart 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:44.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:44.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:44.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:44.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:44.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:44.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:53:44.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:44.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:44.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:44.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:44.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:44.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart 2026-02-22T19:53:44.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:44.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:3 rbd mirror restart 2026-02-22T19:53:44.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:53:44.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:44.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:44.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:44.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:53:44.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:44.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:44.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:44.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:44.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart 2026-02-22T19:53:44.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:53:44.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:44.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:44.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:44.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:53:44.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:44.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:44.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:44.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:53:44.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:44.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:44.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:53:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:45.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:45.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:45.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:53:45.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:45.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:53:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:45.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:45.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:45.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:45.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:45.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:45.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:45.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:45.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:45.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:45.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:45.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:53:45.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:53:45.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:45.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:45.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:53:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:53:45.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:45.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:45.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:45.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:45.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:45.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:53:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:53:45.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:45.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:53:45.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:53:45.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:53:45.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:45.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-22T19:53:45.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:45.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:45.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:45.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-22T19:53:45.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:45.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:45.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:45.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:53:45.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:45.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:45.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:46.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-22T19:53:46.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:46.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:46.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:46.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:53:46.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:46.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-22T19:53:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:53:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:46.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:46.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:46.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:46.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:46.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:46.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:46.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:46.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:46.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:46.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:46.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:46.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:46.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:46.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:53:46.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:53:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:53:46.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:46.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:53:46.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:53:46.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:53:46.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:46.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:53:46.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:46.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:46.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:47.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:53:47.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:47.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:47.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:47.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:47.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:47.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:47.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:47.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:47.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:53:47.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:53:47.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:53:47.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:53:47.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:53:47.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:53:47.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:44' 2026-02-22T19:53:47.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:53:47.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:53:47.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:47.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:53:47.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:47.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:47.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:47.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:47.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:47.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:53:47.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:47.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:44' 2026-02-22T19:53:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:53:44' 2026-02-22T19:53:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:44' 2026-02-22T19:53:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:47.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:47.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:53:44test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:53:44' 2026-02-22T19:53:47.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:47.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:53:47.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:44' 2026-02-22T19:53:47.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:53:47.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:53:47.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:47.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:53:47.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:47.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:47.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:47.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2026-02-22T19:53:47.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:47.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:47.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:47.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:47.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:47.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:47.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:47.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:48.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2026-02-22T19:53:48.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:48.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:48.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:48.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:48.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-22T19:53:48.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test1: 2026-02-22T19:53:48.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:53:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:53:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"} 2026-02-22T19:53:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:53:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:44' 2026-02-22T19:53:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:53:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:53:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:48.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"} 2026-02-22T19:53:48.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:48.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:48.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:48.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:48.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:48.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:53:48.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:48.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:44' 2026-02-22T19:53:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-22 19:53:44' 2026-02-22T19:53:48.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:44' 2026-02-22T19:53:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:48.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:53:44test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:53:44' 2026-02-22T19:53:48.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:48.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-22T19:53:48.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:44' 2026-02-22T19:53:48.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-22T19:53:48.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:53:48.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:48.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:53:48.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:48.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ all_admin_daemons cluster1 rbd mirror stop 2026-02-22T19:53:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:48.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:53:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror stop 2026-02-22T19:53:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:53:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:48.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:48.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:48.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:53:48.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:48.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:48.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:53:48.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:48.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:48.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:48.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:48.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:48.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:48.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop 2026-02-22T19:53:49.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror stop 2026-02-22T19:53:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:53:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:53:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:49.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:49.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:49.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:49.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:49.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop 2026-02-22T19:53:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:2 rbd mirror stop 2026-02-22T19:53:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:53:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:53:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:53:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:53:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:49.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:49.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:53:49.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:49.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:49.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:49.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:53:49.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:49.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:49.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:49.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:49.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:49.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:49.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:49.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:53:49.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:49.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:49.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:49.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:49.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:49.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:3 rbd mirror stop 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:53:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:53:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:53:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:49.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:49.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:49.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:53:49.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:49.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:49.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:49.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:49.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:49.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop 2026-02-22T19:53:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:53:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:53:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:49.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:53:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:49.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:50.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:53:50.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:50.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:50.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:50.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:53:50.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:50.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:50.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:53:50.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:50.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:50.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:50.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:50.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:50.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:50.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:50.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:50.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:50.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:50.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:50.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:50.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:50.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:50.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:50.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:50.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:50.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:50.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:50.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:50.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:50.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:50.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:53:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:53:50.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:50.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:50.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:50.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:50.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:50.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:50.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:50.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:50.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:53:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:50.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:50.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:50.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:50.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:53:50.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:53:50.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:50.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:50.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:50.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:50.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:50.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:50.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:53:50.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:53:50.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:50.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:53:50.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2026-02-22T19:53:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:53:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:51.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:51.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-22T19:53:51.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:51.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:51.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:51.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:51.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:51.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:51.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:51.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:51.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:51.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:53:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:51.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:51.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:51.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:51.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:51.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:51.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:51.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:51.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:51.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:51.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:53:51.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:51.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:53:51.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:53:51.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:51.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:53:51.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:51.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:53:51.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:51.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:51.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:51.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:51.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:51.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:51.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:51.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:52.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:53:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:52.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:52.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:53:52.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:53:52.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:53:52.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:53:52.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: stopped 2026-02-22T19:53:52.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:53:52.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:49' 2026-02-22T19:53:52.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:53:52.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:53:52.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:52.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: stopped 2026-02-22T19:53:52.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:52.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:53:52.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:52.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:53:52.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:52.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:53:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:52.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:49' 2026-02-22T19:53:52.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:53:49' 2026-02-22T19:53:52.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:49' 2026-02-22T19:53:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:52.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+stoppedstopped43424353mirror.1trial0172026-02-22 19:53:49test1f4ee5839-bc21-4b89-840a-30463c6a4063up+stoppedstopped43384346mirror.0trial0172026-02-22 19:53:49' 2026-02-22T19:53:52.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:52.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:53:52.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:49' 2026-02-22T19:53:52.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:53:52.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:53:52.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:53:52.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:53:52.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:53:52.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:52.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:52.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2026-02-22T19:53:52.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:52.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:52.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:52.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:52.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:52.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:52.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:52.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2026-02-22T19:53:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:53.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-22T19:53:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test1: 2026-02-22T19:53:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:53:53.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:53:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: stopped 2026-02-22T19:53:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:53:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:49' 2026-02-22T19:53:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:53:53.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:53:53.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: stopped 2026-02-22T19:53:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:53:53.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:53.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:53:53.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:53.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:53:53.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:53.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:49' 2026-02-22T19:53:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2026-02-22 19:53:49' 2026-02-22T19:53:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:53.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:53:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:49' 2026-02-22T19:53:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:53.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+stoppedstopped43424353mirror.1trial0172026-02-22 19:53:49test1f4ee5839-bc21-4b89-840a-30463c6a4063up+stoppedstopped43384346mirror.0trial0172026-02-22 19:53:49' 2026-02-22T19:53:53.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:53.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-22T19:53:53.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:49' 2026-02-22T19:53:53.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-22T19:53:53.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:53:53.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:53:53.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:53:53.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:53:53.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:53.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:53.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ all_admin_daemons cluster1 rbd mirror restart 2026-02-22T19:53:53.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:53.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:53.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:53:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror restart 2026-02-22T19:53:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:53:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:53.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:53.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:53:53.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:53.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:53.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:53.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:53:53.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:53.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:53.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:53.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:53:53.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:53.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:53.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:53:53.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:53.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:53.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:53.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:53.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:53.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:53.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:53.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:53.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:53.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:53.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:53.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:53.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:53.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart 2026-02-22T19:53:53.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror restart 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:53.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:53.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:53.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:53.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:53.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:53.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart 2026-02-22T19:53:53.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:53.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:2 rbd mirror restart 2026-02-22T19:53:53.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:53:53.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:53.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:53.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:53.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:53.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:53.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:53:53.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:53.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:53.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:53.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart 2026-02-22T19:53:53.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:53.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:3 rbd mirror restart 2026-02-22T19:53:53.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:53:53.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:53.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:53.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:53:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:53.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:53:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:53:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:53.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:53:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:53:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:53.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:53.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:53.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:53.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:53.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:53.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:53.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:53:53.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:53.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:53.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:53.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:53.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:53.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:53.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:53:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:53:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:54.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:53:54.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:54.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:54.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:54.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:54.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:54.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:54.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:54.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:54.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:54.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:54.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:54.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:54.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:54.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:54.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:54.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:54.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:54.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:54.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:54.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:54.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:54.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:54.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:54.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:55.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:53:55.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:53:55.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:55.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:55.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:55.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:55.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:55.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:55.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:55.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:55.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:53:55.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:53:55.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:55.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:55.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:55.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:55.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:55.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:55.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:53:55.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:53:55.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:55.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:53:55.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:53:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:53:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-22T19:53:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-22T19:53:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:55.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:53:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:56.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-22T19:53:56.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:56.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:56.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:56.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:53:56.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:56.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:56.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-22T19:53:56.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:56.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:56.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:56.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:56.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:56.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:56.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:56.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:56.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:56.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:56.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:56.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:53:56.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:56.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:56.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:56.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:56.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:56.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:56.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:56.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:56.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:56.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:56.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:56.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:53:56.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:56.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:53:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:53:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:56.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:53:56.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:53:56.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:53:56.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:56.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:53:56.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:56.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:56.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:56.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:56.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:56.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:56.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:56.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:57.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:53:57.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:57.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:57.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:57.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:57.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:57.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:57.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:57.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:57.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:53:57.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:53:57.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:53:57.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:53:57.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:53:57.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:53:57.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:53' 2026-02-22T19:53:57.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:53:57.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:53:57.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:57.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:53:57.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:57.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:57.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:57.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:57.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:53:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:57.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:57.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:53' 2026-02-22T19:53:57.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:53:53' 2026-02-22T19:53:57.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:57.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:57.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:57.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:57.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:53' 2026-02-22T19:53:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:57.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:53:53test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:53:53' 2026-02-22T19:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:53:57.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:53' 2026-02-22T19:53:57.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:53:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:53:57.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:57.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:53:57.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:57.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:57.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:57.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2026-02-22T19:53:57.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:57.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:57.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:57.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:57.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:57.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:57.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:57.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2026-02-22T19:53:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:53:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:53:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-22T19:53:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test1: 2026-02-22T19:53:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:53:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:53:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"} 2026-02-22T19:53:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:53:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:53' 2026-02-22T19:53:58.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:53:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:53:58.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:53:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"} 2026-02-22T19:53:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:53:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:58.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:53:58.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:58.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:53:58.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:53:58.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:53:58.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:53:58.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:53' 2026-02-22T19:53:58.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-22 19:53:53' 2026-02-22T19:53:58.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:58.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:58.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:58.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:53:58.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:53' 2026-02-22T19:53:58.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:53:58.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:53:58.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:53:58.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:53:53test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:53:53' 2026-02-22T19:53:58.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:53:58.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-22T19:53:58.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:53' 2026-02-22T19:53:58.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-22T19:53:58.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:53:58.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:53:58.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:53:58.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:53:58.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:58.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:58.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2-DNE 2026-02-22T19:53:58.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:58.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:58.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2-DNE 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:58.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:58.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:58.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:58.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:58.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:58.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2-DNE 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2-DNE 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:53:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:58.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:58.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:58.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:58.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:58.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2-DNE 2026-02-22T19:53:58.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:58.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:2 rbd mirror stop mirror cluster2-DNE 2026-02-22T19:53:58.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:53:58.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:58.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:58.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:53:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:53:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:53:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:53:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:53:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:53:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:53:58.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop mirror cluster2-DNE 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:3 rbd mirror stop mirror cluster2-DNE 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:53:58.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop mirror cluster2-DNE 2026-02-22T19:53:58.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:53:58.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:58.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:53:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:53:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:58.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:53:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:53:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:53:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:53:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:53:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:53:59.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:53:59.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:53:59.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:53:59.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:59.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:53:59.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:53:59.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:53:59.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:59.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:53:59.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:53:59.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:53:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:53:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:53:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:53:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:53:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:59.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:53:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:53:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:53:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:53:59.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:53:59.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:59.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:59.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:53:59.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:53:59.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:53:59.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:53:59.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:53:59.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:53:59.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:53:59.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:53:59.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:53:59.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:53:59.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:53:59.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:53:59.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:53:59.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:53:59.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:53:59.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:53:59.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:53:59.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:53:59.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:53:59.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:53:59.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:53:59.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:53:59.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:53:59.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:53:59.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:53:59.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:53:59.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:59.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:59.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:53:59.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:53:59.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:53:59.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:53:59.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:53:59.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:53:59.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:53:59.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:53:59.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2026-02-22T19:53:59.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:59.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2026-02-22T19:53:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:53:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:53:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:53:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:53:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:53:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:53:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:00.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2026-02-22T19:54:00.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:00.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:00.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:00.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:54:00.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:00.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:00.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-22T19:54:00.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:00.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:00.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:00.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:00.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:00.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:00.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:00.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:00.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:00.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:00.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:00.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:00.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:00.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:00.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:00.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:54:00.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:00.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:54:00.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:54:00.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:00.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:54:00.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:00.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:54:00.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:00.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:00.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:00.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:54:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:54:01.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:54:01.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:54:01.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:54:01.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: stopped 2026-02-22T19:54:01.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:54:01.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:58' 2026-02-22T19:54:01.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:54:01.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:54:01.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:01.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: stopped 2026-02-22T19:54:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:54:01.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:54:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:01.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:54:01.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:01.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:01.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:58' 2026-02-22T19:54:01.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:53:58' 2026-02-22T19:54:01.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:01.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:01.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:01.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:01.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:58' 2026-02-22T19:54:01.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:01.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:01.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:01.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+stoppedstopped43424353mirror.1trial0172026-02-22 19:53:58test1f4ee5839-bc21-4b89-840a-30463c6a4063up+stoppedstopped43384346mirror.0trial0172026-02-22 19:53:58' 2026-02-22T19:54:01.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:01.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:58' 2026-02-22T19:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:54:01.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:54:01.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:54:01.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:54:01.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:54:01.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:01.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:01.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2026-02-22T19:54:01.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:01.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:01.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:01.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:01.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:01.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:01.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:01.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:02.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2026-02-22T19:54:02.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:02.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:02.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:02.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:02.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:03.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:03.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:03.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:03.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-22T19:54:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test1: 2026-02-22T19:54:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:54:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:54:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: stopped 2026-02-22T19:54:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:54:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:53:58' 2026-02-22T19:54:03.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:54:03.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:54:03.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:03.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: stopped 2026-02-22T19:54:03.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:03.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:54:03.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:03.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:54:03.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:03.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:54:03.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:03.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:03.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:58' 2026-02-22T19:54:03.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2026-02-22 19:53:58' 2026-02-22T19:54:03.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:03.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:03.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:03.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:03.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:53:58' 2026-02-22T19:54:03.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:03.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:03.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:03.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+stoppedstopped43424353mirror.1trial0172026-02-22 19:53:58test1f4ee5839-bc21-4b89-840a-30463c6a4063up+stoppedstopped43384346mirror.0trial0172026-02-22 19:53:58' 2026-02-22T19:54:03.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:03.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-22T19:54:03.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:53:58' 2026-02-22T19:54:03.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-22T19:54:03.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:54:03.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:54:03.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:54:03.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:54:03.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:03.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:03.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2026-02-22T19:54:03.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:03.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:03.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:03.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance=cluster1 2026-02-22T19:54:03.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loop_instance 2026-02-22T19:54:03.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:03.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:03.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:03.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:54:03.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:03.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2026-02-22T19:54:03.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:03.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:03.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:03.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:03.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:03.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:03.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:03.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:03.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:03.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2026-02-22T19:54:03.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:54:03.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror start mirror/test1 2026-02-22T19:54:03.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:03.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror start mirror/test 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:03.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:03.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:03.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:03.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start mirror/test 2026-02-22T19:54:03.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:03.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:54:03.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:03.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:03.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:03.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:54:03.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:03.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:03.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:03.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:03.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:03.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:03.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:54:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:04.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:04.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:04.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:04.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:04.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:04.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:04.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:04.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:04.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:04.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:04.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:04.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:04.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:04.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:04.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:04.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:04.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:04.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:04.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:04.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:04.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:04.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:04.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:04.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:04.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:04.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:54:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:54:04.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:04.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:04.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:54:04.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:54:04.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:04.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:04.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:04.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:04.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:04.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:04.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:04.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:04.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:04.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:04.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:04.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:04.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:04.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:04.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:54:04.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:04.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:04.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:04.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:54:04.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:04.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:04.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:04.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:04.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:04.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:04.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:04.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:54:04.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:54:04.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:04.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:04.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:04.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:04.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:54:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:04.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:04.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:04.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:54:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:04.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:05.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:54:05.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:05.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:54:05.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:54:05.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:54:05.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:54:05.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:54:05.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:54:05.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:03' 2026-02-22T19:54:05.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:54:05.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:54:05.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:05.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:54:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:05.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:54:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:05.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:03' 2026-02-22T19:54:05.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:54:03' 2026-02-22T19:54:05.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:05.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:05.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:05.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:05.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:03' 2026-02-22T19:54:05.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:05.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:05.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:05.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:54:03test1f4ee5839-bc21-4b89-840a-30463c6a4063up+stoppedstopped43384346mirror.0trial0172026-02-22 19:53:58' 2026-02-22T19:54:05.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:05.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:54:05.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:03' 2026-02-22T19:54:05.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:54:05.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:54:05.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:05.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:54:05.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:05.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ all_admin_daemons cluster1 rbd mirror start mirror cluster2-DNE 2026-02-22T19:54:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:05.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:54:05.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror cluster2-DNE 2026-02-22T19:54:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:54:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:54:05.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:05.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:05.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror cluster2-DNE 2026-02-22T19:54:05.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:05.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror start mirror cluster2-DNE 2026-02-22T19:54:05.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:54:05.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:05.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:05.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:05.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:54:05.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:05.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:05.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:05.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:54:05.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start mirror cluster2-DNE 2026-02-22T19:54:05.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:05.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:2 rbd mirror start mirror cluster2-DNE 2026-02-22T19:54:05.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:54:05.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:05.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:05.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:54:05.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:54:05.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:05.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:05.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:54:05.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:54:05.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:05.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:05.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:05.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:05.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:05.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:05.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:05.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:54:05.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:54:05.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:54:05.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror start mirror cluster2-DNE 2026-02-22T19:54:05.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:05.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:3 rbd mirror start mirror cluster2-DNE 2026-02-22T19:54:05.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:54:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:54:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:54:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:54:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:05.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:05.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:05.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:05.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:05.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:05.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:54:05.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:54:05.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:54:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:05.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror start mirror cluster2-DNE 2026-02-22T19:54:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-22T19:54:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-22T19:54:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:06.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-22T19:54:06.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-22T19:54:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:06.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:06.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:06.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:06.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:06.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:54:06.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:06.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:06.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:06.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:06.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:06.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:06.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:54:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:07.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:07.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:07.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:07.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:07.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:07.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:07.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:07.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2026-02-22T19:54:07.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:07.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:07.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:07.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:07.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:07.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:07.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:07.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2026-02-22T19:54:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:08.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-22T19:54:08.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test1: 2026-02-22T19:54:08.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:54:08.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:54:08.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"} 2026-02-22T19:54:08.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:54:08.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:05' 2026-02-22T19:54:08.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:54:08.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:54:08.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:08.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"} 2026-02-22T19:54:08.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:08.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:08.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:08.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:08.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:05' 2026-02-22T19:54:08.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-22 19:54:05' 2026-02-22T19:54:08.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:08.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:08.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:08.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:08.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:05' 2026-02-22T19:54:08.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:08.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:08.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:08.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:54:03test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:54:05' 2026-02-22T19:54:08.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:08.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-22T19:54:08.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:05' 2026-02-22T19:54:08.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-22T19:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:54:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test 2026-02-22T19:54:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance=cluster1 2026-02-22T19:54:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loop_instance 2026-02-22T19:54:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:08.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:08.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:08.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:54:08.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:08.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test 2026-02-22T19:54:08.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:08.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:08.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:08.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:08.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:54:08.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:08.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:08.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:08.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:08.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:08.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:08.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:08.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test 2026-02-22T19:54:08.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:54:08.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror restart mirror/test1 2026-02-22T19:54:08.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:08.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:08.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror/test 2026-02-22T19:54:08.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:54:08.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:08.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:08.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:08.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:08.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:08.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:08.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:08.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:08.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror/test 2026-02-22T19:54:08.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:09.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:54:09.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:09.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:09.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:09.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:09.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:09.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:09.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:09.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:09.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:09.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:09.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:09.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:09.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:09.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:54:09.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:54:09.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:09.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:09.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:09.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:09.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:09.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:54:09.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:09.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:09.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:09.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:09.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:54:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:54:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:54:10.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:54:10.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:54:10.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:54:10.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:54:10.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:54:10.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:08' 2026-02-22T19:54:10.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:54:10.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:54:10.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:10.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:54:10.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:10.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:10.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:10.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:10.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:54:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:10.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:08' 2026-02-22T19:54:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:54:08' 2026-02-22T19:54:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:08' 2026-02-22T19:54:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:10.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:10.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:54:08test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:54:05' 2026-02-22T19:54:10.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:10.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:54:10.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:08' 2026-02-22T19:54:10.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:54:10.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:54:10.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:10.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:54:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:54:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:54:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:10.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:10.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:10.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:10.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:10.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:10.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:10.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:54:10.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:54:10.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:10.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:10.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:10.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:10.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:10.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:10.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:10.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:10.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:10.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:2 rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:10.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:10.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:10.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:54:10.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:54:10.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:54:10.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:10.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:10.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:10.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:3 rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:10.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:54:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:54:10.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:10.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:10.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:10.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:10.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:54:10.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:10.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:10.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:10.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:54:10.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:10.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:10.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:10.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:10.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:10.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:10.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:54:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:11.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:11.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:11.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:11.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:11.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:11.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:11.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:11.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:11.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:11.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:11.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:11.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:11.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:11.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:11.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:11.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:11.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:11.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:11.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:11.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:11.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:11.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:11.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:11.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:11.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:11.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:11.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:11.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:11.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:11.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:11.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:11.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:11.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:11.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:11.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:11.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:11.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:11.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:11.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:11.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:54:11.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:54:11.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:11.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:11.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:11.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:11.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:11.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:11.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:54:11.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:54:11.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:11.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:11.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:11.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:11.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:54:11.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:11.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-22T19:54:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:11.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-22T19:54:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:11.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:12.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-22T19:54:12.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:12.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:12.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-22T19:54:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:12.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:12.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:12.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:12.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:12.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:12.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:12.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:12.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:54:12.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:12.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:13.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:54:13.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:13.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:13.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:13.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:13.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:13.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2-DNE 2026-02-22T19:54:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:13.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:54:13.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2-DNE 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:13.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:13.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:13.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:13.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:13.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:13.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:13.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:13.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:13.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:13.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2-DNE 2026-02-22T19:54:13.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:13.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2-DNE 2026-02-22T19:54:13.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:54:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:54:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:54:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2-DNE 2026-02-22T19:54:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:2 rbd mirror stop mirror cluster2-DNE 2026-02-22T19:54:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:54:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:54:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:54:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:54:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:54:13.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:13.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop mirror cluster2-DNE 2026-02-22T19:54:13.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:13.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:3 rbd mirror stop mirror cluster2-DNE 2026-02-22T19:54:13.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:54:13.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:13.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:54:13.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:54:13.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:13.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:13.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:54:13.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:54:13.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:13.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:13.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:13.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:13.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:13.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:13.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:13.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:54:13.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:54:13.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:54:13.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:13.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:13.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop mirror cluster2-DNE 2026-02-22T19:54:13.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:54:13.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:13.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:13.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:13.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:54:13.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:13.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:13.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:13.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:54:13.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:13.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:13.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:54:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:54:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:14.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:14.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:14.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:14.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:14.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:14.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:14.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:14.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:14.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:14.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:14.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:14.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:14.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:14.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:14.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:14.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:14.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:14.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:14.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:14.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:14.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:14.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:14.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:14.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:14.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:14.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:14.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:54:14.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:54:14.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:14.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:14.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:14.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:14.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:14.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:14.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:14.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:14.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:54:14.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:54:14.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:14.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:14.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:14.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:14.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:54:14.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:14.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:54:14.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:54:14.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:14.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:15.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2026-02-22T19:54:15.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:15.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:15.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:15.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:54:15.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:15.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:15.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-22T19:54:15.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:15.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:15.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:15.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:15.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:15.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:15.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:15.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:54:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:15.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:15.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:15.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:15.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:54:15.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:15.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:54:15.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:54:15.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:15.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:54:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:54:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:15.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:16.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:54:16.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:16.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:16.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:16.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:16.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:16.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:16.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:16.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:16.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:54:16.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:54:16.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:54:16.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:54:16.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: stopped 2026-02-22T19:54:16.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:54:16.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:13' 2026-02-22T19:54:16.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:54:16.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:54:16.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:16.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: stopped 2026-02-22T19:54:16.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:16.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:54:16.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:16.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:54:16.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:16.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:54:16.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:16.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:13' 2026-02-22T19:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:54:13' 2026-02-22T19:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:13' 2026-02-22T19:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:16.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+stoppedstopped43424353mirror.1trial0172026-02-22 19:54:13test1f4ee5839-bc21-4b89-840a-30463c6a4063up+stoppedstopped43384346mirror.0trial0172026-02-22 19:54:13' 2026-02-22T19:54:16.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:16.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:54:16.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:13' 2026-02-22T19:54:16.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:54:16.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:54:16.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:54:16.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:54:16.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:54:16.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:16.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:16.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2026-02-22T19:54:16.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:16.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:16.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:16.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:16.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:16.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:16.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:16.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2026-02-22T19:54:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:17.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-22T19:54:17.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test1: 2026-02-22T19:54:17.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:54:17.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:54:17.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: stopped 2026-02-22T19:54:17.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:54:17.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:13' 2026-02-22T19:54:17.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:54:17.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:54:17.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:17.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: stopped 2026-02-22T19:54:17.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:17.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:54:17.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:17.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:54:17.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:17.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:54:17.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:17.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:17.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:13' 2026-02-22T19:54:17.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2026-02-22 19:54:13' 2026-02-22T19:54:17.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:17.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:17.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:17.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:54:17.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:13' 2026-02-22T19:54:17.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:17.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:17.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+stoppedstopped43424353mirror.1trial0172026-02-22 19:54:13test1f4ee5839-bc21-4b89-840a-30463c6a4063up+stoppedstopped43384346mirror.0trial0172026-02-22 19:54:13' 2026-02-22T19:54:17.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:17.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-22T19:54:17.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:13' 2026-02-22T19:54:17.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-22T19:54:17.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:54:17.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:54:17.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:54:17.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:54:17.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:17.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:17.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:17.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:17.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:17.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:54:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:54:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:17.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:17.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:17.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:17.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:17.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:17.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:17.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:54:17.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:17.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:17.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:17.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:54:17.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:17.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:17.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:17.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:17.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:2 rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:17.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:54:17.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:17.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:17.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:54:17.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:54:17.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:17.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:17.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:17.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:54:17.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:17.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:17.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:17.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:54:17.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:17.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:17.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:54:17.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:54:17.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:54:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:3 rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:54:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:54:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:54:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:54:17.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:54:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:54:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:54:17.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:54:18.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:18.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:18.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:18.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart mirror cluster2-DNE 2026-02-22T19:54:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:54:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:54:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:18.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:18.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:18.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:18.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:19.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:54:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:19.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:19.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:19.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:19.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:19.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:19.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:19.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:19.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:19.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:19.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:19.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:19.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:19.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:19.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:19.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:19.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:19.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:no valid command found; 1 closest matches: 2026-02-22T19:54:19.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status mirror/test1 2026-02-22T19:54:19.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:54:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:54:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:54:19.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:54:19.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:54:19.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:19.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:19.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:19.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:19.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:54:19.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:19.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:19.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:19.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:19.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:19.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:19.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:19.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:19.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2026-02-22T19:54:19.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:19.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:19.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:19.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2026-02-22T19:54:19.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:19.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:19.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:19.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:19.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:19.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:19.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:20.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test1 started 2026-02-22T19:54:20.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:20.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:20.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:20.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:20.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:20.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2026-02-22T19:54:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:20.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:20.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:20.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2026-02-22T19:54:20.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:20.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:20.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:20.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:20.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:20.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:20.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:20.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:20.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:20.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2026-02-22T19:54:20.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:20.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:20.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:20.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:20.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:20.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:21.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:54:21.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:21.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:21.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:21.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:21.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:21.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:54:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:54:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:54:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:54:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:54:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:54:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:17' 2026-02-22T19:54:21.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:54:21.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:54:21.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:21.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:54:21.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:21.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:21.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:21.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:21.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:21.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:54:21.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:21.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:21.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:17' 2026-02-22T19:54:21.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:54:17' 2026-02-22T19:54:21.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:21.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:21.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:21.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:21.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:17' 2026-02-22T19:54:21.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:21.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:21.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:54:17test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:54:17' 2026-02-22T19:54:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:21.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:54:21.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:17' 2026-02-22T19:54:21.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:54:21.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:54:21.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:21.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:54:21.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2026-02-22T19:54:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:21.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:21.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:22.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2026-02-22T19:54:22.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:22.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:22.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:22.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:22.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:22.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:22.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:22.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:22.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2026-02-22T19:54:22.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test1: 2026-02-22T19:54:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: f4ee5839-bc21-4b89-840a-30463c6a4063 2026-02-22T19:54:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:54:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"} 2026-02-22T19:54:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:54:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:17' 2026-02-22T19:54:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:54:22.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:54:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"} 2026-02-22T19:54:22.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:22.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:22.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:22.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:22.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:22.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:54:22.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:22.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:17' 2026-02-22T19:54:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2026-02-22 19:54:17' 2026-02-22T19:54:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:17' 2026-02-22T19:54:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:22.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:22.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:22.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtestd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:54:17test1f4ee5839-bc21-4b89-840a-30463c6a4063up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790006,"remote_snapshot_timestamp":1771790006,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:54:17' 2026-02-22T19:54:22.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:22.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2026-02-22T19:54:22.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:17' 2026-02-22T19:54:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2026-02-22T19:54:22.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:54:22.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:22.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:54:22.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:22.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:22.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:22.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ flush cluster1 2026-02-22T19:54:22.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:22.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool= 2026-02-22T19:54:22.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image= 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'cmd=rbd mirror flush' 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemons cluster1 rbd mirror flush 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance=cluster1 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loop_instance 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:22.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:54:22.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:22.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror flush 2026-02-22T19:54:22.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:22.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:22.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:22.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:22.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:22.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush 2026-02-22T19:54:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ all_admin_daemons cluster1 rbd mirror status 2026-02-22T19:54:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:22.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:22.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:54:22.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:22.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror status 2026-02-22T19:54:22.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:22.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:22.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:22.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:22.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:54:22.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:22.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:22.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:22.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:54:22.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:22.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:22.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:22.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:54:22.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:22.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:22.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:22.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:22.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:22.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:22.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:22.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:22.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:22.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status 2026-02-22T19:54:22.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:{ 2026-02-22T19:54:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool_replayers": [ 2026-02-22T19:54:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "peer": "uuid: c42be08c-54e0-47b4-8faa-b0889455bae6 cluster: cluster2-DNE client: client.mirror", 2026-02-22T19:54:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool": "mirror", 2026-02-22T19:54:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "instance_id": "4346", 2026-02-22T19:54:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "stopped (manual)", 2026-02-22T19:54:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader_instance_id": "4346", 2026-02-22T19:54:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader": true, 2026-02-22T19:54:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "instances": [ 2026-02-22T19:54:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "4346", 2026-02-22T19:54:22.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "4353", 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "4358", 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "4361" 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ], 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.21766.cluster1.94322405620072.asok", 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.21766.cluster2-DNE.94322418424168.asok", 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "sync_throttler": { 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 5, 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "deletion_throttler": { 2026-02-22T19:54:22.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 1, 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [ 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "mirror/test1", 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ], 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_deleter": { 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_deleter_status": { 2026-02-22T19:54:22.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "delete_images_queue": [], 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "failed_deletes_queue": [] 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "namespaces": [ 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns1", 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [], 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_deleter": { 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_deleter_status": { 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "delete_images_queue": [], 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "failed_deletes_queue": [] 2026-02-22T19:54:22.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns2", 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [], 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_deleter": { 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_deleter_status": { 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "delete_images_queue": [], 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "failed_deletes_queue": [] 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "peer": "uuid: 4a3ca826-0c80-43e0-bfc9-77783b303376 cluster: cluster2-DNE client: client.mirror", 2026-02-22T19:54:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool": "mirror_parent", 2026-02-22T19:54:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "instance_id": "4372", 2026-02-22T19:54:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "stopped (manual)", 2026-02-22T19:54:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader_instance_id": "4371", 2026-02-22T19:54:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader": false, 2026-02-22T19:54:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.21766.cluster1.94322406209896.asok", 2026-02-22T19:54:22.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.21766.cluster2-DNE.94322408282472.asok", 2026-02-22T19:54:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "sync_throttler": { 2026-02-22T19:54:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 5, 2026-02-22T19:54:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "deletion_throttler": { 2026-02-22T19:54:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 1, 2026-02-22T19:54:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [], 2026-02-22T19:54:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "namespaces": [ 2026-02-22T19:54:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns1", 2026-02-22T19:54:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [] 2026-02-22T19:54:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:} 2026-02-22T19:54:22.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror status 2026-02-22T19:54:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:54:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:54:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:22.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:22.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:22.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:22.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:22.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:22.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status 2026-02-22T19:54:22.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:{ 2026-02-22T19:54:22.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool_replayers": [ 2026-02-22T19:54:22.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "peer": "uuid: c42be08c-54e0-47b4-8faa-b0889455bae6 cluster: cluster2-DNE client: client.mirror", 2026-02-22T19:54:22.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool": "mirror", 2026-02-22T19:54:22.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "instance_id": "4353", 2026-02-22T19:54:22.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "stopped (manual)", 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader_instance_id": "4346", 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader": false, 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.21794.cluster1.94289455888744.asok", 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.21794.cluster2-DNE.94289468635496.asok", 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "sync_throttler": { 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 5, 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "deletion_throttler": { 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 1, 2026-02-22T19:54:22.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [ 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "mirror/test", 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ], 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "namespaces": [ 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns1", 2026-02-22T19:54:22.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [] 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns2", 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [] 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "peer": "uuid: 4a3ca826-0c80-43e0-bfc9-77783b303376 cluster: cluster2-DNE client: client.mirror", 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool": "mirror_parent", 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "instance_id": "4365", 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "stopped (manual)", 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader_instance_id": "4371", 2026-02-22T19:54:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader": false, 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.21794.cluster1.94289456372072.asok", 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.21794.cluster2-DNE.94289458329960.asok", 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "sync_throttler": { 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 5, 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "deletion_throttler": { 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 1, 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [], 2026-02-22T19:54:22.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "namespaces": [ 2026-02-22T19:54:22.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns1", 2026-02-22T19:54:22.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [] 2026-02-22T19:54:22.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:} 2026-02-22T19:54:22.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:22.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:2 rbd mirror status 2026-02-22T19:54:22.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T19:54:22.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:22.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:22.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:54:22.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T19:54:22.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:22.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:22.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:22.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:22.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:54:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:54:22.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:54:22.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:22.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:22.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:22.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status 2026-02-22T19:54:22.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:{ 2026-02-22T19:54:22.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool_replayers": [ 2026-02-22T19:54:22.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "peer": "uuid: c42be08c-54e0-47b4-8faa-b0889455bae6 cluster: cluster2-DNE client: client.mirror", 2026-02-22T19:54:22.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool": "mirror", 2026-02-22T19:54:22.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "instance_id": "4358", 2026-02-22T19:54:22.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "stopped (manual)", 2026-02-22T19:54:22.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader_instance_id": "4346", 2026-02-22T19:54:22.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader": false, 2026-02-22T19:54:22.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.2.21837.cluster1.94673119082856.asok", 2026-02-22T19:54:22.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.21837.cluster2-DNE.94673131960680.asok", 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "sync_throttler": { 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 5, 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "deletion_throttler": { 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 1, 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [], 2026-02-22T19:54:22.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "namespaces": [ 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns1", 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [] 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns2", 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [] 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "peer": "uuid: 4a3ca826-0c80-43e0-bfc9-77783b303376 cluster: cluster2-DNE client: client.mirror", 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool": "mirror_parent", 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "instance_id": "4373", 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "stopped (manual)", 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader_instance_id": "4371", 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader": false, 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.2.21837.cluster1.94673134303592.asok", 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.21837.cluster2-DNE.94673121581416.asok", 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "sync_throttler": { 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 5, 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "deletion_throttler": { 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 1, 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [], 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "namespaces": [ 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns1", 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [] 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:} 2026-02-22T19:54:22.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:22.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:3 rbd mirror status 2026-02-22T19:54:22.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T19:54:22.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:22.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:22.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T19:54:22.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:54:22.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:54:22.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:54:22.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:22.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:22.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:22.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status 2026-02-22T19:54:22.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:{ 2026-02-22T19:54:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool_replayers": [ 2026-02-22T19:54:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "peer": "uuid: c42be08c-54e0-47b4-8faa-b0889455bae6 cluster: cluster2-DNE client: client.mirror", 2026-02-22T19:54:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool": "mirror", 2026-02-22T19:54:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "instance_id": "4361", 2026-02-22T19:54:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "stopped (manual)", 2026-02-22T19:54:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader_instance_id": "4346", 2026-02-22T19:54:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader": false, 2026-02-22T19:54:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.3.21889.cluster1.94873105897832.asok", 2026-02-22T19:54:22.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.21889.cluster2-DNE.94873118775656.asok", 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "sync_throttler": { 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 5, 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "deletion_throttler": { 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 1, 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [], 2026-02-22T19:54:22.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "namespaces": [ 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns1", 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [] 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns2", 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [] 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "peer": "uuid: 4a3ca826-0c80-43e0-bfc9-77783b303376 cluster: cluster2-DNE client: client.mirror", 2026-02-22T19:54:22.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "pool": "mirror_parent", 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "instance_id": "4371", 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "stopped (manual)", 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader_instance_id": "4371", 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "leader": true, 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "instances": [ 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "4365", 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "4371", 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "4372", 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "4373" 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ], 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.3.21889.cluster1.94873106356584.asok", 2026-02-22T19:54:22.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.21889.cluster2-DNE.94873108306280.asok", 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "sync_throttler": { 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 5, 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "deletion_throttler": { 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "max_parallel_requests": 1, 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "running_requests": 0, 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "waiting_requests": 0 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [], 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_deleter": { 2026-02-22T19:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_deleter_status": { 2026-02-22T19:54:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "delete_images_queue": [], 2026-02-22T19:54:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "failed_deletes_queue": [] 2026-02-22T19:54:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: }, 2026-02-22T19:54:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "namespaces": [ 2026-02-22T19:54:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: { 2026-02-22T19:54:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "ns1", 2026-02-22T19:54:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_replayers": [], 2026-02-22T19:54:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_deleter": { 2026-02-22T19:54:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "image_deleter_status": { 2026-02-22T19:54:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "delete_images_queue": [], 2026-02-22T19:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "failed_deletes_queue": [] 2026-02-22T19:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: } 2026-02-22T19:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: ] 2026-02-22T19:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:} 2026-02-22T19:54:22.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test1 2026-02-22T19:54:22.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:22.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T19:54:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test1 2026-02-22T19:54:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test1 2026-02-22T19:54:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test1 2026-02-22T19:54:22.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test1 2026-02-22T19:54:24.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T19:54:24.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:24.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: test image rename' 2026-02-22T19:54:24.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T19:54:24.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T19:54:24.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 19:54:24 TEST: test image rename 2026-02-22T19:54:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 19:54:24 TEST: test image rename 2026-02-22T19:54:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ new_name=test_RENAMED 2026-02-22T19:54:24.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_image cluster2 mirror test test_RENAMED 2026-02-22T19:54:24.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:24.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:24.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:24.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_name=test_RENAMED 2026-02-22T19:54:24.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rename mirror/test mirror/test_RENAMED 2026-02-22T19:54:24.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:54:24.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test_RENAMED 2026-02-22T19:54:24.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:24.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:24.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:24.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test_RENAMED 2026-02-22T19:54:26.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 8 2026-02-22T19:54:26.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED 2026-02-22T19:54:26.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:26.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:26.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:26.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started 2026-02-22T19:54:26.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:26.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:26.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:26.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:26.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:26.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:26.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:27.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started 2026-02-22T19:54:27.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:27.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:27.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:27.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:27.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:27.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:27.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:27.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:27.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:27.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:27.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:27.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:27.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:27.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:27.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:27.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:27.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:27.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:27.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:27.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:27.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2026-02-22T19:54:27.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:27.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:27.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:54:27.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:54:27.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:27.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:27.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:27.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:27.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:27.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:27.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:27.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:27.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:27.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2026-02-22T19:54:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:27.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:27.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:28.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying '' '' 2026-02-22T19:54:28.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:28.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:28.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:28.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:28.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:28.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:28.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:28.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_RENAMED 2026-02-22T19:54:28.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_RENAMED: 2026-02-22T19:54:28.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:54:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:54:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:54:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:54:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:23' 2026-02-22T19:54:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:54:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:54:28.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:28.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"} 2026-02-22T19:54:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:28.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:28.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:28.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:54:28.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:28.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:28.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:23' 2026-02-22T19:54:28.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying '2026-02-22 19:54:23' 2026-02-22T19:54:28.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:28.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:28.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:28.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:23' 2026-02-22T19:54:28.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:28.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:28.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:28.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_RENAMEDd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790012,"remote_snapshot_timestamp":1771790012,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:54:23' 2026-02-22T19:54:28.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:28.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/last_update' 2026-02-22T19:54:28.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:23' 2026-02-22T19:54:28.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/state' 2026-02-22T19:54:28.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:54:28.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:28.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:54:28.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:28.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:28.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:28.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:28.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:28.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:28.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance=cluster1 2026-02-22T19:54:28.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loop_instance 2026-02-22T19:54:28.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:28.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:28.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:28.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:28.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2026-02-22T19:54:28.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:28.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:28.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:28.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:28.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:54:28.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:28.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:28.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:28.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:54:28.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:28.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:28.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:28.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:28.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:28.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:28.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:28.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:28.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:28.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:28.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:28.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:28.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:28.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:28.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:28.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:28.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:28.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:28.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:{ 2026-02-22T19:54:28.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "name": "mirror/test_RENAMED", 2026-02-22T19:54:28.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:28.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:} 2026-02-22T19:54:28.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:28.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test_RENAMED 2026-02-22T19:54:28.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_instance=cluster1 2026-02-22T19:54:28.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:28.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance=cluster1 2026-02-22T19:54:28.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loop_instance 2026-02-22T19:54:28.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:28.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:28.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:28.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:54:28.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:28.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test_RENAMED 2026-02-22T19:54:28.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T19:54:28.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:28.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:28.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:28.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T19:54:28.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:28.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:28.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:28.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:54:28.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:28.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:28.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:28.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test_RENAMED 2026-02-22T19:54:28.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2026-02-22T19:54:28.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror restart : restart rbd mirror 2026-02-22T19:54:28.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:28.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:28.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror/test_RENAMED 2026-02-22T19:54:28.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T19:54:28.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:28.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:28.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:28.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T19:54:28.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:28.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:28.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T19:54:28.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:54:28.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:28.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T19:54:28.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:28.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:54:28.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:28.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:28.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:28.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:28.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:28.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:28.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:28.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror/test_RENAMED 2026-02-22T19:54:28.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:28.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED 2026-02-22T19:54:28.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started 2026-02-22T19:54:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:29.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started 2026-02-22T19:54:29.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:29.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:29.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:29.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:29.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:29.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:29.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:29.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:29.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:29.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:29.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2026-02-22T19:54:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:54:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:54:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:54:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:29.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:29.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:29.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:29.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2026-02-22T19:54:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:29.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:29.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2026-02-22T19:54:29.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:29.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:30.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying '' '' 2026-02-22T19:54:30.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:30.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:30.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:30.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:30.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:30.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:30.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:30.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:30.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_RENAMED 2026-02-22T19:54:30.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_RENAMED: 2026-02-22T19:54:30.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: d3facf73-8492-44d6-bde1-34803b6b4fd6 2026-02-22T19:54:30.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:54:30.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790065,"remote_snapshot_timestamp":1771790065,"replay_state":"idle"} 2026-02-22T19:54:30.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T19:54:30.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:28' 2026-02-22T19:54:30.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:54:30.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:54:30.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:30.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790065,"remote_snapshot_timestamp":1771790065,"replay_state":"idle"} 2026-02-22T19:54:30.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:30.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:30.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:30.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:30.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:30.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T19:54:30.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:30.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:30.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:28' 2026-02-22T19:54:30.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying '2026-02-22 19:54:28' 2026-02-22T19:54:30.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:30.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:28' 2026-02-22T19:54:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:30.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_RENAMEDd3facf73-8492-44d6-bde1-34803b6b4fd6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790065,"remote_snapshot_timestamp":1771790065,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 19:54:28' 2026-02-22T19:54:30.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:30.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/last_update' 2026-02-22T19:54:30.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:28' 2026-02-22T19:54:30.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/state' 2026-02-22T19:54:30.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:54:30.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:30.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:54:30.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:30.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:30.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:30.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_image cluster2 mirror test_RENAMED test 2026-02-22T19:54:30.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:30.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:30.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_RENAMED 2026-02-22T19:54:30.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_name=test 2026-02-22T19:54:30.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rename mirror/test_RENAMED mirror/test 2026-02-22T19:54:30.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:54:30.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:54:30.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:30.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:30.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:30.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:54:32.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 9 2026-02-22T19:54:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:54:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:54:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:33.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:54:33.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:33.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:33.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:33.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:33.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:33.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:33.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:33.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:33.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:33.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:33.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:33.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:33.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:33.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:33.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:33.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:54:33.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:33.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:33.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:33.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:54:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:54:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:33.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:33.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:33.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:33.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:33.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:54:33.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:54:33.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:54:33.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:33.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:33.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:33.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:33.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:33.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:33.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:33.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:33.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: test trash move restore' 2026-02-22T19:54:33.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T19:54:33.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T19:54:33.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 19:54:33 TEST: test trash move restore 2026-02-22T19:54:33.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 19:54:33 TEST: test trash move restore 2026-02-22T19:54:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster2 mirror test 2026-02-22T19:54:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:54:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:33.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T19:54:33.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster2 info mirror/test 2026-02-22T19:54:33.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T19:54:33.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id=10bb7145ac75 2026-02-22T19:54:33.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ trash_move cluster2 mirror test 2026-02-22T19:54:33.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:33.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:33.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:33.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 trash move mirror/test 2026-02-22T19:54:35.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test deleted 2026-02-22T19:54:35.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:35.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:35.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:35.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T19:54:35.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T19:54:35.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:35.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:54:35.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T19:54:35.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:35.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:35.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T19:54:35.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T19:54:35.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T19:54:35.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T19:54:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id= 2026-02-22T19:54:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T19:54:35.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T19:54:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test deleted '' 2026-02-22T19:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T19:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T19:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T19:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T19:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T19:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T19:54:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T19:54:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T19:54:35.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T19:54:35.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T19:54:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:54:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T19:54:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ trash_restore cluster2 mirror test 10bb7145ac75 snapshot 2026-02-22T19:54:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=10bb7145ac75 2026-02-22T19:54:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T19:54:35.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 trash restore mirror/10bb7145ac75 2026-02-22T19:54:35.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T19:54:35.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:35.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T19:54:35.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:35.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test snapshot 2026-02-22T19:54:35.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:35.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:35.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:35.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T19:54:35.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2026-02-22T19:54:36.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T19:54:36.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test 2026-02-22T19:54:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test': 2026-02-22T19:54:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T19:54:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T19:54:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T19:54:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 10bb7145ac75 2026-02-22T19:54:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.10bb7145ac75 2026-02-22T19:54:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T19:54:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T19:54:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T19:54:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T19:54:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 19:53:11 2026 2026-02-22T19:54:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 19:53:11 2026 2026-02-22T19:54:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 19:53:11 2026 2026-02-22T19:54:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 10bb7145ac75 2026-02-22T19:54:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T19:54:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T19:54:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: e4706b23-e57a-4173-9267-3e7c78859c22 2026-02-22T19:54:36.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T19:54:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:54:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:54:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:37.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:54:37.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:37.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:37.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:37.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:37.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:37.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:37.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:37.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:37.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:37.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:37.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:37.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:37.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:37.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:37.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:37.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:37.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:37.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:37.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:37.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:37.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:37.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:37.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:37.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:37.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:37.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:54:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:37.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:37.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:37.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:54:37.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:54:37.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:37.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:37.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:37.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:37.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:54:37.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:54:37.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:37.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:37.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:37.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T19:54:37.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T19:54:37.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:37.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:37.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:54:37.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:54:37.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:37.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:37.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:54:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T19:54:37.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:37.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:37.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:37.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:54:37.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:37.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:37.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:37.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T19:54:37.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T19:54:37.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:54:37.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:37.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:37.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:37.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T19:54:37.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:54:37.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:37.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:37.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:37.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T19:54:37.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:37.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:54:37.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:37.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:37.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:37.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T19:54:37.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T19:54:37.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:54:37.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:37.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:37.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:37.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T19:54:37.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:54:37.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:37.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:37.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T19:54:38.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:38.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:38.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:38.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:38.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:38.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:38.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:38.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:38.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:38.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:38.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:38.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed (with rbd-mirror on one cluster)' 2026-02-22T19:54:38.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T19:54:38.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T19:54:38.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 19:54:38 TEST: check if removed 'images'\''' OMAP are removed '(with' rbd-mirror on one 'cluster)' 2026-02-22T19:54:38.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 19:54:38 TEST: check if removed images' OMAP are removed (with rbd-mirror on one cluster) 2026-02-22T19:54:38.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test 2026-02-22T19:54:38.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:38.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:38.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:38.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T19:54:38.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:38.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test 2026-02-22T19:54:38.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:38.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:38.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:38.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2026-02-22T19:54:38.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test 2026-02-22T19:54:39.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T19:54:39.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_in_omap cluster1 mirror 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirroring 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=status_global 2026-02-22T19:54:39.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T19:54:39.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:54:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2026-02-22T19:54:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T19:54:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=status_global 2026-02-22T19:54:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-22T19:54:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c status_global 2026-02-22T19:54:39.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:39.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:39.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T19:54:39.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-22T19:54:39.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T19:54:39.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T19:54:39.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:54:39.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T19:54:39.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2026-02-22T19:54:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirroring 2026-02-22T19:54:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=image_ 2026-02-22T19:54:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T19:54:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:39.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:54:39.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2026-02-22T19:54:39.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:39.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:39.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T19:54:39.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=image_ 2026-02-22T19:54:39.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-22T19:54:39.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:39.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c image_ 2026-02-22T19:54:39.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:39.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T19:54:39.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-22T19:54:39.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T19:54:39.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T19:54:39.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:54:39.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T19:54:39.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2026-02-22T19:54:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirror_leader 2026-02-22T19:54:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=image_map 2026-02-22T19:54:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T19:54:39.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:39.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:54:39.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-22T19:54:39.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:39.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:39.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T19:54:39.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=image_map 2026-02-22T19:54:39.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-22T19:54:39.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:39.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:39.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T19:54:39.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-22T19:54:39.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c image_map 2026-02-22T19:54:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 1 = 0 2026-02-22T19:54:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=1 2026-02-22T19:54:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:54:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 -eq 0 ']' 2026-02-22T19:54:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T19:54:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:40.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:54:40.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-22T19:54:40.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:40.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:40.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T19:54:40.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=image_map 2026-02-22T19:54:40.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-22T19:54:40.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c image_map 2026-02-22T19:54:40.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:40.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:40.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T19:54:40.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-22T19:54:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 1 = 0 2026-02-22T19:54:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=1 2026-02-22T19:54:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:54:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 -eq 0 ']' 2026-02-22T19:54:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T19:54:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:54:42.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:54:42.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-22T19:54:42.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:42.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:42.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T19:54:42.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=image_map 2026-02-22T19:54:42.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-22T19:54:42.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:42.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c image_map 2026-02-22T19:54:42.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:42.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T19:54:42.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-22T19:54:42.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T19:54:42.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T19:54:42.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:54:42.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T19:54:42.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:42.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_in_omap cluster2 mirror 2026-02-22T19:54:42.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:42.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:42.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2026-02-22T19:54:42.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:42.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:42.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirroring 2026-02-22T19:54:42.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=status_global 2026-02-22T19:54:42.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T19:54:42.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:42.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:54:42.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2026-02-22T19:54:42.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:42.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T19:54:42.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=status_global 2026-02-22T19:54:42.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-22T19:54:42.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:42.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T19:54:42.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-22T19:54:42.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c status_global 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirroring 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=image_ 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:54:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2026-02-22T19:54:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T19:54:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=image_ 2026-02-22T19:54:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-22T19:54:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T19:54:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-22T19:54:42.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c image_ 2026-02-22T19:54:42.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T19:54:42.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T19:54:42.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:54:42.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T19:54:42.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:42.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2026-02-22T19:54:42.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:42.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:42.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirror_leader 2026-02-22T19:54:42.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=image_map 2026-02-22T19:54:42.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T19:54:42.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T19:54:42.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T19:54:42.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2026-02-22T19:54:42.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:42.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T19:54:42.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=image_map 2026-02-22T19:54:42.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2026-02-22T19:54:42.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:54:42.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T19:54:42.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T19:54:42.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2026-02-22T19:54:42.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c image_map 2026-02-22T19:54:42.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:error getting omap key set mirror/rbd_mirror_leader: (2) No such file or directory 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test 2026-02-22T19:54:42.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T19:54:42.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:42.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T19:54:42.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:42.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test snapshot 2026-02-22T19:54:42.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:42.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:42.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:42.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T19:54:42.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2026-02-22T19:54:43.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T19:54:43.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test 2026-02-22T19:54:43.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test': 2026-02-22T19:54:43.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T19:54:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T19:54:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T19:54:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 11514e1103e6 2026-02-22T19:54:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.11514e1103e6 2026-02-22T19:54:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T19:54:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T19:54:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T19:54:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T19:54:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 19:54:42 2026 2026-02-22T19:54:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 19:54:42 2026 2026-02-22T19:54:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 19:54:42 2026 2026-02-22T19:54:43.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 11514e1103e6 2026-02-22T19:54:43.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T19:54:43.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T19:54:43.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:54:43.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T19:54:43.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:54:43.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:43.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:43.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:54:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:54:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:44.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:44.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:44.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:44.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:44.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:44.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:54:44.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:44.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T19:54:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:44.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:44.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T19:54:44.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T19:54:44.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T19:54:44.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:44.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:44.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T19:54:44.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-22T19:54:44.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:54:44.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:44.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:44.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:44.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-22T19:54:44.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:44.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T19:54:44.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T19:54:44.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:44.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:44.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T19:54:44.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T19:54:44.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T19:54:44.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:44.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:44.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T19:54:44.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-22T19:54:44.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:54:44.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:44.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-22T19:54:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T19:54:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T19:54:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T19:54:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:44.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:44.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:44.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:44.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:44.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:44.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T19:54:44.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:54:44.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T19:54:44.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T19:54:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T19:54:44.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T19:54:44.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:44.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:44.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T19:54:44.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T19:54:44.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T19:54:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:44.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T19:54:45.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:45.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:45.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:45.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:45.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:45.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:45.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:45.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:45.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:45.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:45.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:45.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:45.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:45.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:45.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:45.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:45.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test 100 2026-02-22T19:54:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T19:54:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T19:54:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:54:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T19:54:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T19:54:45.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T19:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T19:54:45.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 5000.23 bytes/sec: 20 MiB/s 2026-02-22T19:54:45.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:54:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:54:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:54:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:54:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:54:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:45.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:45.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:54:46.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 12 2026-02-22T19:54:46.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:54:46.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:54:46.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:46.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:46.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:46.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:54:46.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:54:46.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:46.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:54:46.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:54:46.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:54:46.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:54:46.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:54:46.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:54:46.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:46.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:46.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:46.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:54:46.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:54:46.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:54:46.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:54:46.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=12 2026-02-22T19:54:46.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:54:46.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:46.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:54:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:54:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:54:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:54:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:54:46.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:54:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=11 2026-02-22T19:54:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 12 = 11 2026-02-22T19:54:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:54:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T19:54:46.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:54:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:54:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:54:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:54:46.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=12 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 12 = 12 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:47.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:54:47.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:47.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:47.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:47.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:47.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:47.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:47.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:47.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:47.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790083,"replay_state":"idle"} 2026-02-22T19:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:45' 2026-02-22T19:54:47.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:54:47.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:47.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790083,"replay_state":"idle"} 2026-02-22T19:54:47.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:47.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:47.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:47.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:47.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:47.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:54:47.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:47.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:47.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:45' 2026-02-22T19:54:47.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:54:45' 2026-02-22T19:54:47.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:47.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:47.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:47.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:54:47.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:45' 2026-02-22T19:54:47.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:47.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:47.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790083,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:54:45' 2026-02-22T19:54:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:47.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:54:47.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:45' 2026-02-22T19:54:47.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:54:47.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:54:47.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:54:47.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:54:47.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:54:47.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:47.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:47.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: failover and failback' 2026-02-22T19:54:47.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T19:54:47.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T19:54:47.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 19:54:47 TEST: failover and failback 2026-02-22T19:54:47.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 19:54:47 TEST: failover and failback 2026-02-22T19:54:47.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirrors cluster2 2026-02-22T19:54:47.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:47.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T19:54:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster2:0 2026-02-22T19:54:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:0 2026-02-22T19:54:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster2:0 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster2 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster2 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:54:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:54:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster2:1 2026-02-22T19:54:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:1 2026-02-22T19:54:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:47.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster2:1 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster2 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster2 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:54:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:54:47.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:47.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster2:2 2026-02-22T19:54:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:2 2026-02-22T19:54:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-22T19:54:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster2:2 2026-02-22T19:54:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster2 2026-02-22T19:54:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T19:54:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-22T19:54:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster2 2026-02-22T19:54:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:54:47.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T19:54:47.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T19:54:47.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:54:47.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster2 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:54:47.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:47.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster2:3 2026-02-22T19:54:47.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:3 2026-02-22T19:54:47.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T19:54:47.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-22T19:54:47.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster2:3 2026-02-22T19:54:47.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T19:54:47.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T19:54:47.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster2 2026-02-22T19:54:47.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T19:54:47.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-22T19:54:47.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster2 2026-02-22T19:54:47.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:54:47.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T19:54:47.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T19:54:47.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:54:47.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster2 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T19:54:47.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T19:54:47.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:47.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:47.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:47.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T19:54:49.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:54:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:54:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:54:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:54:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:49.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:50.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:54:50.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:50.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:50.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:50.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:54:50.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:50.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:50.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:50.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:50.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:50.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:50.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:50.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:50.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:50.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:50.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:50.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:50.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:50.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:50.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:50.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:50.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:54:50.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:54:50.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:50.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:54:50.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:50.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:54:50.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:50.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:50.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:50.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:50.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:51.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:54:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:54:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:51.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:54:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:54:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:54:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:54:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:54:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:54:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:49 2026-02-22T19:54:51.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:54:51.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:54:51.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:54:51.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:54:51.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:50' 2026-02-22T19:54:51.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:54:51.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:54:51.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:51.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:54:51.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:51.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:54:51.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:51.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:54:51.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:51.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:54:51.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:51.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:51.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:49' 2026-02-22T19:54:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:54:49' 2026-02-22T19:54:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:54:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:49' 2026-02-22T19:54:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:51.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 19:54:49cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:54:50' 2026-02-22T19:54:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:51.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:54:51.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:49' 2026-02-22T19:54:51.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:54:51.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:54:51.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:54:51.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:54:51.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:51.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:51.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:54:51.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:51.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:51.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:52.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:54:52.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:52.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:52.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:52.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:54:52.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:52.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:52.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:52.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:52.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:54:52.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:54:52.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:54:52.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:54:52.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:54:52.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:50 2026-02-22T19:54:52.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:54:52.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:54:52.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:54:52.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:54:52.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:49' 2026-02-22T19:54:52.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:54:52.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:54:52.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:52.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:54:52.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:52.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:54:52.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:52.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:54:52.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:52.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T19:54:52.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:52.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:54:54.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:54:54.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:54.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:54.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:54.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:54:54.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:54:54.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:54:54.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:54.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:54.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:54:54.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:54:54.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:54:54.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:54:54.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:54:54.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:54:54.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:52 2026-02-22T19:54:54.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:54:54.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:54:54.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:54:54.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:54:54.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:53' 2026-02-22T19:54:54.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:54:54.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:54:54.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:54:54.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:54:54.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:54:54.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:54:54.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:54:54.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:54:54.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:54:54.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:54:54.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:54:54.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:54:54.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:52' 2026-02-22T19:54:54.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:54:52' 2026-02-22T19:54:54.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:54.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:54.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:54.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:54:54.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:52' 2026-02-22T19:54:54.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:54:54.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:54:54.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:54:54.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:54:52cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:54:53' 2026-02-22T19:54:54.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:54:54.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:54:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:52' 2026-02-22T19:54:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:54:54.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:54:54.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:54:54.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:54:54.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:54:54.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:54.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:54.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T19:54:54.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:54.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:54.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:54.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:54:54.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T19:54:56.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:54:56.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:54:56.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:56.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:56.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:56.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:54:56.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:56.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:56.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:56.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:54:56.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:54:56.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:56.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:54:57.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:54:57.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:57.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:57.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:57.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:57.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:57.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:57.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:57.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:57.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:57.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:57.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:57.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:57.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:57.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:57.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:57.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:57.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:57.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:57.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:57.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:57.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:57.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:57.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:57.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:57.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:57.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:57.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:57.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:57.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:54:57.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:54:57.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:57.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T19:54:57.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:57.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:54:59.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:54:59.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:54:59.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:59.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:59.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:54:59.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:54:59.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:54:59.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:54:59.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:54:59.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:54:59.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:54:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:54:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:54:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:54:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:54:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:54:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:54:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:54:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:54:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:54:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:54:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:54:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:54:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:54:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:54:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:54:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:54:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:54:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:54:59.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:54:59.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:54:59.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:54:59.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:54:59.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:54:59.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:54:59.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:54:59.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:54:59.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:54:59.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:54:59.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:54:59.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:54:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:54:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:54:59.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:54:59.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:54:59.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:54:59.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:54:59.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:54:59.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test 100 2026-02-22T19:54:59.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:59.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:59.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:59.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T19:54:59.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T19:54:59.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:54:59.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T19:54:59.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T19:54:59.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T19:54:59.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T19:54:59.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 5000.18 bytes/sec: 20 MiB/s 2026-02-22T19:54:59.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:54:59.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:54:59.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:54:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:54:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:54:59.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:54:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:54:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:54:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:55:01.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 15 2026-02-22T19:55:01.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:55:01.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:55:01.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:01.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:01.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:01.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:01.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:55:01.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:55:01.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:55:01.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:01.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:01.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:01.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:01.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:01.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:01.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:01.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:55:01.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:55:01.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:55:01.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=15 2026-02-22T19:55:01.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:55:01.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:55:01.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:01.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:01.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:01.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:01.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:01.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:55:01.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:55:01.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:55:01.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=15 2026-02-22T19:55:01.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 15 = 15 2026-02-22T19:55:01.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:01.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T19:55:01.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:01.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:55:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:01.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:02.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T19:55:02.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:02.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:02.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:02.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:55:02.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:02.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:02.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:02.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:02.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:55:02.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:02.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:02.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:55:02.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:55:02.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:02.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:57 2026-02-22T19:55:02.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:02.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:55:02.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:55:02.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790088,"remote_snapshot_timestamp":1771790095,"replay_state":"idle"} 2026-02-22T19:55:02.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:58 2026-02-22T19:55:02.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:55:02.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 14 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.e2029794-584a-4d77-9d60-fdc7358c74be (peer_uuids:[]) 2026-02-22T19:55:02.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 15 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.11743478-b975-48c6-a10b-587854b4238f (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:55:02.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:55:02.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:55:02.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:02.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:55:02.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:02.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:55:02.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:02.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:55:02.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:57' 2026-02-22T19:55:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 19:54:57' 2026-02-22T19:55:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:57' 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:02.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 19:54:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790088,"remote_snapshot_timestamp":1771790095,"replay_state":"idle"}2026-02-22 19:54:58' 2026-02-22T19:55:02.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:02.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:02.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:57' 2026-02-22T19:55:02.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:02.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:55:02.468 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:55:02.468 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:55:02.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-22T19:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:03.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:55:03.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:03.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:03.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:03.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:03.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:03.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:03.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:03.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:03.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:55:03.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:03.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:03.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:55:03.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790088,"remote_snapshot_timestamp":1771790095,"replay_state":"idle"} 2026-02-22T19:55:03.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:03.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:54:58 2026-02-22T19:55:03.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:03.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:55:03.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:55:03.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:55:03.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:02' 2026-02-22T19:55:03.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:55:03.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:55:03.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:03.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790088,"remote_snapshot_timestamp":1771790095,"replay_state":"idle"} 2026-02-22T19:55:03.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:03.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:55:03.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:03.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:55:03.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:03.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:03.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:03.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:03.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:58' 2026-02-22T19:55:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:54:58' 2026-02-22T19:55:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:54:58' 2026-02-22T19:55:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:03.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790088,"remote_snapshot_timestamp":1771790095,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:54:58cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:55:02' 2026-02-22T19:55:03.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:03.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:03.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:54:58' 2026-02-22T19:55:03.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:03.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:55:03.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:55:03.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:55:03.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:55:03.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:03.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test 2026-02-22T19:55:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T19:55:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.export 2026-02-22T19:55:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.export 2026-02-22T19:55:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:03.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:55:03.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.export 2026-02-22T19:55:03.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.export 2026-02-22T19:55:03.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:03.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:55:03.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:55:03.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:03.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:55:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:03.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:03.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T19:55:03.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:03.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:03.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:03.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T19:55:05.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:55:05.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:55:05.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:05.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:05.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:05.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:55:05.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:05.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:05.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:05.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:55:05.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:55:05.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:05.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:06.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:55:06.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:06.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:06.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:06.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:55:06.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:06.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:06.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:55:06.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:06.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:55:06.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:06.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:55:06.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:55:06.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:06.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:06.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:06.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:55:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:06.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:06.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:06.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:06.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:06.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:55:06.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:55:06.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:06.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:55:06.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:06.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:55:06.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:06.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:06.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:06.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:06.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:06.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:06.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:06.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:07.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:55:07.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:07.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:07.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:07.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:07.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:07.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:07.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:07.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:07.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:55:07.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:07.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:07.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:07.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:55:07.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:07.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:05 2026-02-22T19:55:07.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:07.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:55:07.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:55:07.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:55:07.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:02' 2026-02-22T19:55:07.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:55:07.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:55:07.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:07.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:55:07.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:07.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:07.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:07.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:07.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:07.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:07.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:07.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:07.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:05' 2026-02-22T19:55:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:55:05' 2026-02-22T19:55:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:05' 2026-02-22T19:55:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:07.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:07.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 19:55:05cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:55:02' 2026-02-22T19:55:07.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:07.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:07.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:05' 2026-02-22T19:55:07.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:07.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:55:07.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:07.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:55:07.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:07.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:07.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:07.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:55:07.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:07.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:07.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:07.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:07.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:07.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:07.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:07.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:55:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:08.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:08.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:08.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:08.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:08.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:08.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:08.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:55:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:55:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:07 2026-02-22T19:55:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:55:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:55:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:08' 2026-02-22T19:55:08.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:55:08.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:55:08.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:08.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:55:08.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:08.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:08.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:08.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:08.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:08.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:08.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:08.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:07' 2026-02-22T19:55:08.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:55:07' 2026-02-22T19:55:08.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:08.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:08.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:08.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:08.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:07' 2026-02-22T19:55:08.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:08.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:08.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:08.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:55:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:55:08' 2026-02-22T19:55:08.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:08.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:08.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:07' 2026-02-22T19:55:08.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:08.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:55:08.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:08.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:55:08.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:08.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:08.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:08.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T19:55:08.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:08.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:08.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:08.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:55:08.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T19:55:08.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:55:08.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T19:55:08.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:08.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:08.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:08.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T19:55:08.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:08.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:08.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:08.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:55:08.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:55:08.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:08.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:09.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:55:09.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:09.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:09.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:09.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:55:09.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:09.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:09.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:55:09.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:09.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:55:09.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:09.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:55:09.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:55:09.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:09.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:09.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:09.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:55:09.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:55:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:55:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:55:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:55:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:55:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:55:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:55:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:09.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:55:09.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:09.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:09.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:09.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:09.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:09.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:55:09.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:55:09.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:09.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T19:55:09.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:09.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:55:11.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:55:11.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:11.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:11.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:11.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:55:11.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:11.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:11.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:55:11.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:11.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:55:11.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:11.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:55:11.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:55:11.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:11.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:11.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:11.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:55:11.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:55:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:55:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:55:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:55:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:55:11.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:55:11.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:11.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:11.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:11.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:55:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:55:11.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:11.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T19:55:11.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:11.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 4 2026-02-22T19:55:15.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:55:15.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:15.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:15.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:15.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:55:15.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:15.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:55:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:55:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:55:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:55:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:15.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:15.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:55:15.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:15.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:15.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:15.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:15.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:15.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:15.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:15.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:55:15.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:55:15.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:15.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:15.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:55:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:15.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:15.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:55:15.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:55:15.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:15.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:55:15.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:55:15.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:55:15.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:15.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T19:55:15.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:15.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:15.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:15.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T19:55:19.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:55:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T19:55:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T19:55:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:55:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:55:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:19.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:55:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:20.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:55:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:20.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:20.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:55:20.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:20.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:55:20.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:20.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:55:20.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:55:20.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:20.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:20.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:20.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:55:20.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:20.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:20.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:55:20.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:55:20.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:20.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:20.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:55:20.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:55:20.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:20.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:20.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:55:20.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:20.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:20.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:20.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:20.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:20.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:20.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:55:20.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:55:20.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:20.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:20.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:55:20.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:20.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:20.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:20.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:20.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:20.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:55:20.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:55:20.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:20.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:55:20.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:20.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:55:20.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:20.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:20.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:20.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:20.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:20.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:20.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:20.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:55:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:55:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:55:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:18 2026-02-22T19:55:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:55:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:55:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:18' 2026-02-22T19:55:21.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:55:21.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:55:21.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:21.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:55:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:21.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:21.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:21.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:21.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:21.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:21.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:18' 2026-02-22T19:55:21.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:55:18' 2026-02-22T19:55:21.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:21.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:21.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:21.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:21.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:18' 2026-02-22T19:55:21.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:21.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:21.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:21.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:55:18cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 19:55:18' 2026-02-22T19:55:21.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:21.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:21.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:18' 2026-02-22T19:55:21.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:21.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:55:21.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:21.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:55:21.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:21.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:21.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:21.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:55:21.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:21.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:21.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:21.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:21.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:21.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:21.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:21.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:22.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:55:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:22.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:55:22.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:22.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:22.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:22.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:55:22.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:22.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:18 2026-02-22T19:55:22.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:22.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:55:22.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:22.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:55:22.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:18' 2026-02-22T19:55:22.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:55:22.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:55:22.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:22.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:55:22.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:22.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:22.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:22.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:22.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:22.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:22.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:22.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:22.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:18' 2026-02-22T19:55:22.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:55:18' 2026-02-22T19:55:22.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:22.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:18' 2026-02-22T19:55:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:22.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 19:55:18cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:55:18' 2026-02-22T19:55:22.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:22.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:18' 2026-02-22T19:55:22.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:22.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:55:22.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:22.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:55:22.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:22.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:22.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:22.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T19:55:22.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:22.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:22.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:22.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:55:22.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T19:55:24.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:55:24.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:55:24.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:24.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:24.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:24.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:55:24.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:24.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:24.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:24.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:55:24.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:55:24.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:24.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:25.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:55:25.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:25.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:25.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:25.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:55:25.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:25.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:55:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:55:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:55:25.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:55:25.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:25.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:25.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:25.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:25.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:25.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:25.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:55:25.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:55:25.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:25.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:25.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:25.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:55:25.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:55:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:25.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:25.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:25.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:25.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:25.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:25.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:25.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:55:25.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:55:25.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:25.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:55:25.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:55:25.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:55:25.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:25.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:55:25.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:55:25.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:25.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:25.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:25.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:55:25.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:55:25.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:55:25.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:25.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:25.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:25.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:55:29.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 20 2026-02-22T19:55:29.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:55:29.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:55:29.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:29.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:29.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:29.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:29.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:55:29.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:29.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:29.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:55:29.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:55:29.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:29.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:29.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:29.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:55:29.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:55:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:55:29.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=20 2026-02-22T19:55:29.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:55:29.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:29.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:55:29.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:55:29.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:55:29.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:55:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=20 2026-02-22T19:55:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 20 = 20 2026-02-22T19:55:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-22T19:55:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:55:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:55:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:30.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:55:30.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:30.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:30.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:30.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:30.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:30.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:30.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:30.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:30.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:55:30.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:30.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:30.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:55:30.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790123,"replay_state":"idle"} 2026-02-22T19:55:30.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:30.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:23 2026-02-22T19:55:30.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:30.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:55:30.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:55:30.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:55:30.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:27' 2026-02-22T19:55:30.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:55:30.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:55:30.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:30.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790123,"replay_state":"idle"} 2026-02-22T19:55:30.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:30.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:55:30.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:30.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:55:30.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:30.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:30.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:30.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:30.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:23' 2026-02-22T19:55:30.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:55:23' 2026-02-22T19:55:30.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:30.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:30.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:30.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:30.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:23' 2026-02-22T19:55:30.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:30.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:30.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:30.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790123,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:55:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:55:27' 2026-02-22T19:55:30.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:30.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:30.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:23' 2026-02-22T19:55:30.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:55:30.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:55:30.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:55:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:55:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T19:55:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:55:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:31.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T19:55:31.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:55:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:31.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:55:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:55:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:55:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:27 2026-02-22T19:55:31.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:31.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:55:31.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:55:31.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790123,"replay_state":"idle"} 2026-02-22T19:55:31.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:23 2026-02-22T19:55:31.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:55:31.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 20 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.16f2c230-8a10-4410-92ea-3614cbb3e836 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:55:31.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:55:31.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:55:31.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:31.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:55:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:55:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:55:31.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:31.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:31.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:31.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:27' 2026-02-22T19:55:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 19:55:27' 2026-02-22T19:55:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:55:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:27' 2026-02-22T19:55:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:31.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:31.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 19:55:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790123,"replay_state":"idle"}2026-02-22 19:55:23' 2026-02-22T19:55:31.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:31.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:31.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:27' 2026-02-22T19:55:31.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:55:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:55:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:55:31.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:55:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test 2026-02-22T19:55:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T19:55:31.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.export 2026-02-22T19:55:31.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.export 2026-02-22T19:55:31.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:55:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.export 2026-02-22T19:55:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.export 2026-02-22T19:55:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:31.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:55:31.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:55:31.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:31.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:55:31.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:31.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:31.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T19:55:31.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:31.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:31.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:31.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T19:55:32.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:55:32.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:55:32.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:32.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:32.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:32.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:55:32.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:32.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:32.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:32.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:55:32.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:55:32.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:32.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:55:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:33.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:55:33.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:33.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:33.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:55:33.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:33.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:55:33.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:33.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:55:33.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:55:33.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:33.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:33.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:33.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:33.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:33.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:33.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:55:33.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:55:33.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:33.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:33.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:33.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:55:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:33.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:33.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:33.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:33.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:33.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:33.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:55:33.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:55:33.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:33.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:55:33.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:55:33.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:55:33.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:33.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:55:35.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:55:35.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:35.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:35.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:35.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:55:35.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:35.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:35.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:55:35.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:35.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:55:35.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:35.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:55:35.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:55:35.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:35.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:35.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:35.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:35.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:35.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:35.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:55:35.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:55:35.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:35.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:35.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:35.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:55:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:55:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:35.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:35.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:35.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:35.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:35.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:35.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:55:35.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:55:35.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:36.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:55:36.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:36.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:36.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:36.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:36.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:36.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:36.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:36.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:36.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:55:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:55:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:33 2026-02-22T19:55:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:55:36.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:36.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:55:36.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:32' 2026-02-22T19:55:36.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:55:36.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:55:36.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:55:36.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:36.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:36.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:36.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:36.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:36.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:36.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:36.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:36.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:33' 2026-02-22T19:55:36.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:55:33' 2026-02-22T19:55:36.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:36.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:36.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:36.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:36.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:33' 2026-02-22T19:55:36.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:36.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 19:55:33cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:55:32' 2026-02-22T19:55:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:36.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:33' 2026-02-22T19:55:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:36.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:55:36.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:36.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:55:36.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:36.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:36.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:36.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:55:36.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:36.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:36.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:36.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:55:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:37.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:37.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:55:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:55:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:32 2026-02-22T19:55:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:55:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:55:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:33' 2026-02-22T19:55:37.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:55:37.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:55:37.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:37.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:55:37.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:37.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:37.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:37.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:37.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:32' 2026-02-22T19:55:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:55:32' 2026-02-22T19:55:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:32' 2026-02-22T19:55:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:37.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:37.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:37.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:55:32cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 19:55:33' 2026-02-22T19:55:37.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:37.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:37.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:32' 2026-02-22T19:55:37.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:37.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:55:37.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:37.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:55:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T19:55:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:55:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T19:55:37.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:55:37.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T19:55:37.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:37.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:37.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:37.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T19:55:37.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:37.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:37.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:37.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:55:37.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:55:37.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:37.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:38.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:55:38.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:38.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:38.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:38.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:55:38.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:38.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:38.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:55:38.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:38.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:55:38.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:38.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:55:38.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:55:38.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:38.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:38.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:38.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:55:38.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:38.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:55:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:55:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:55:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:55:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:55:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:38.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:55:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:55:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:55:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:55:38.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:55:38.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:38.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:38.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:38.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:55:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:55:38.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:38.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster1 mirror test 100 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T19:55:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T19:55:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T19:55:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T19:55:38.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 5000.08 bytes/sec: 20 MiB/s 2026-02-22T19:55:38.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T19:55:38.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:55:38.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:38.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:38.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:38.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:55:38.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:55:38.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T19:55:38.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:38.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:38.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:38.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T19:55:42.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 23 2026-02-22T19:55:42.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T19:55:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:55:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:55:42.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:55:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:42.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:42.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:55:42.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:42.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:42.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:55:42.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:55:42.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:55:42.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=23 2026-02-22T19:55:42.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:55:42.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:42.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:55:42.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:42.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:42.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:42.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:42.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:42.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:55:42.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:55:43.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:55:43.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=23 2026-02-22T19:55:43.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 23 = 23 2026-02-22T19:55:43.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:43.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:55:43.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:43.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:43.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:43.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:55:43.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:43.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:43.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:43.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:44.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:55:44.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:44.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:44.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:44.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:55:44.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:44.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:44.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:44.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:44.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:55:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:55:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:55:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:43 2026-02-22T19:55:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:55:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:55:44.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790137,"replay_state":"idle"} 2026-02-22T19:55:44.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:37 2026-02-22T19:55:44.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:55:44.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 23 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.5a5f2019-fb55-4229-a324-f6dbc4906549 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T19:55:44.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:55:44.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:55:44.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:44.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:55:44.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:44.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:55:44.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:44.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:55:44.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:44.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:44.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:44.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:44.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:43' 2026-02-22T19:55:44.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:55:43' 2026-02-22T19:55:44.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:44.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:44.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:44.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:55:44.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:43' 2026-02-22T19:55:44.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:44.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:44.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:44.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 19:55:43cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790137,"replay_state":"idle"}2026-02-22 19:55:37' 2026-02-22T19:55:44.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:44.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:44.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:43' 2026-02-22T19:55:44.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:44.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:55:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:55:44.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:55:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:55:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster2 mirror test 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T19:55:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:55:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:55:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790137,"replay_state":"idle"} 2026-02-22T19:55:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:37 2026-02-22T19:55:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:55:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:55:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:55:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:43' 2026-02-22T19:55:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:55:45.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:55:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:45.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790137,"replay_state":"idle"} 2026-02-22T19:55:45.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:45.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:55:45.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:45.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:55:45.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:45.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:45.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:45.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:45.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:37' 2026-02-22T19:55:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 19:55:37' 2026-02-22T19:55:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:37' 2026-02-22T19:55:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:45.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790137,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 19:55:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 19:55:43' 2026-02-22T19:55:45.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:45.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:45.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:37' 2026-02-22T19:55:45.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:45.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:55:45.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:55:45.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:55:45.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:55:45.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:45.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:45.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test 2026-02-22T19:55:45.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:45.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:45.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T19:55:45.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.export 2026-02-22T19:55:45.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.export 2026-02-22T19:55:45.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:45.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:55:45.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.export 2026-02-22T19:55:45.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.export 2026-02-22T19:55:45.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:45.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:45.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:45.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:55:45.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:55:45.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:45.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:55:45.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:45.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:55:45.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:45.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T19:55:45.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:45.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:45.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:45.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T19:55:45.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:55:45.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T19:55:45.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:45.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:45.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:45.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T19:55:45.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:45.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:45.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:45.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:55:45.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:55:45.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:45.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:55:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:55:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:46.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:55:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:55:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:55:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:55:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:46.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:55:46.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:46.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:55:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:55:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:55:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:55:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:55:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:46.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:46.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:55:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:55:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:46.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:55:46.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:46.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:46.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:46.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:46.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:46.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:55:46.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:55:46.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:46.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:55:46.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:55:46.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:55:46.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:46.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:55:48.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:55:48.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:48.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:48.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:48.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:55:48.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:48.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:48.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:55:48.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:48.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:55:48.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:48.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:55:48.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:55:48.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:48.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:48.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:55:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:48.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:48.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:55:48.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:48.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:55:48.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:55:48.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:55:48.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:48.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:48.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:55:48.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:48.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:48.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:55:48.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:48.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:48.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:55:48.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:55:48.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:48.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:55:48.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:48.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:55:48.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:48.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:48.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:48.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:48.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:48.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:48.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:48.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:49.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:55:49.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:49.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:49.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:49.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:49.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:49.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:49.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:49.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:49.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:55:49.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:55:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:48 2026-02-22T19:55:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:55:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:55:49.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:47' 2026-02-22T19:55:49.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:55:49.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:55:49.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:49.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:55:49.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:49.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:49.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:49.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:49.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:49.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:49.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:49.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:48' 2026-02-22T19:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:55:48' 2026-02-22T19:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:48' 2026-02-22T19:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:50.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:55:48cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:55:47' 2026-02-22T19:55:50.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:50.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:50.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:48' 2026-02-22T19:55:50.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:50.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:55:50.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:50.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:55:50.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:50.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:50.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:50.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:55:50.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:50.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:50.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:50.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:50.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:50.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:50.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:50.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:51.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:55:51.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:51.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:51.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:51.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:51.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:51.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:51.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:51.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:51.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:55:51.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:55:51.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:55:51.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:51.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:55:51.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:55:51.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:47 2026-02-22T19:55:51.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:55:51.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:55:51.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:55:51.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:55:51.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:48' 2026-02-22T19:55:51.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:55:51.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:55:51.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:55:51.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:55:51.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:55:51.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:51.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:55:51.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:51.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:55:51.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:55:51.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:55:51.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:55:51.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:47' 2026-02-22T19:55:51.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:55:47' 2026-02-22T19:55:51.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:51.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:51.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:51.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:55:51.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:47' 2026-02-22T19:55:51.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:55:51.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:55:51.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:55:51.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:55:47cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:55:48' 2026-02-22T19:55:51.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:55:51.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:55:51.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:47' 2026-02-22T19:55:51.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:55:51.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:55:51.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:55:51.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T19:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:55:51.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T19:55:52.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:55:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:55:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:55:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:52.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:52.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:52.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:55:52.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:55:52.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:52.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:55:53.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:55:53.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:53.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:53.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:53.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:55:53.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:53.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:53.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:55:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:55:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:55:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:55:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:53.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:53.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:53.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:53.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:55:53.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:53.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:53.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:55:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:55:53.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:53.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T19:55:53.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:53.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:55:55.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:55:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:55:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:55:55.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:55:55.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:55:55.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:55:55.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:55:55.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:55.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:55:55.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:55:55.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:55:55.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:55.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:55:55.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:55:55.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:55:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:55:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:55:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:55:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:55:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:55:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:55:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:55:55.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:55:55.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:55:55.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:55:55.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:55:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:55:55.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:55:55.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:55:55.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:55:55.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:55:55.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:55:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:55:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:55:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test 100 2026-02-22T19:55:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T19:55:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T19:55:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T19:55:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T19:55:55.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T19:55:55.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T19:55:55.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T19:55:55.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 5000.08 bytes/sec: 20 MiB/s 2026-02-22T19:55:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:55:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:55:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:55:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:55:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:55:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:55:59.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 26 2026-02-22T19:55:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:55:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:55:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:59.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:55:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:55:59.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:59.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:59.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:55:59.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:55:59.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:55:59.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:59.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:59.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:55:59.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:59.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:59.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:55:59.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:55:59.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:55:59.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:55:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=26 2026-02-22T19:55:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:55:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:55:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:55:59.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:59.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:59.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:59.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:59.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:55:59.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:55:59.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:55:59.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:55:59.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=26 2026-02-22T19:55:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 26 = 26 2026-02-22T19:55:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:55:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-22T19:55:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:55:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:55:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:55:59.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:55:59.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:55:59.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:55:59.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:55:59.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:55:59.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:55:59.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:00.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:56:00.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:00.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:00.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:00.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:00.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:00.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:00.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:00.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:00.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:56:00.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:00.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:00.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:56:00.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790151,"replay_state":"idle"} 2026-02-22T19:56:00.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:00.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:53 2026-02-22T19:56:00.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:00.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:56:00.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:56:00.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:56:00.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:57' 2026-02-22T19:56:00.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:56:00.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:56:00.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:00.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790151,"replay_state":"idle"} 2026-02-22T19:56:00.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:00.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:56:00.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:56:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:00.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:00.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:00.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:53' 2026-02-22T19:56:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:55:53' 2026-02-22T19:56:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:53' 2026-02-22T19:56:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:00.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790151,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:55:53cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:55:57' 2026-02-22T19:56:00.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:00.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:53' 2026-02-22T19:56:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:00.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:56:00.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:56:00.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:56:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:56:00.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:00.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:00.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T19:56:00.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:00.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:00.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:00.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:00.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T19:56:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:56:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:56:01.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:57 2026-02-22T19:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790151,"replay_state":"idle"} 2026-02-22T19:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:55:53 2026-02-22T19:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 26 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.ca3db77f-12fb-40c8-925f-bd588ede7a50 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:56:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:56:01.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:01.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:56:01.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:01.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:56:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:56:01.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:01.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:01.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:01.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:57' 2026-02-22T19:56:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 19:55:57' 2026-02-22T19:56:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:55:57' 2026-02-22T19:56:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:01.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 19:55:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790151,"replay_state":"idle"}2026-02-22 19:55:53' 2026-02-22T19:56:01.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:01.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:01.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:55:57' 2026-02-22T19:56:01.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:01.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:56:01.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:56:01.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:56:01.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:56:01.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:01.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:01.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test 2026-02-22T19:56:01.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:01.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:01.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T19:56:01.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.export 2026-02-22T19:56:01.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.export 2026-02-22T19:56:01.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:01.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:56:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.export 2026-02-22T19:56:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.export 2026-02-22T19:56:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:56:01.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:56:01.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T19:56:01.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:56:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:56:01.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T19:56:01.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:56:01.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T19:56:01.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:01.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: failover / failback loop' 2026-02-22T19:56:01.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T19:56:01.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T19:56:01.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 19:56:01 TEST: failover / failback loop 2026-02-22T19:56:01.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 19:56:01 TEST: failover / failback loop 2026-02-22T19:56:01.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 1 20 2026-02-22T19:56:01.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T19:56:01.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T19:56:01.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:01.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:01.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:01.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T19:56:02.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:03.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:56:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:56:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:56:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:56:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:03.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:56:03.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:56:03.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:03.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:03.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:03.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:03.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:03.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:03.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:56:03.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:56:03.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:03.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:56:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:03.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:03.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:03.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:03.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:56:03.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:56:03.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:03.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:56:03.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:56:03.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:56:03.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:03.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:56:05.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:56:05.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:05.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:05.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:05.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:56:05.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:05.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:05.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:56:05.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:05.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:56:05.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:05.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:56:05.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:56:05.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:05.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:05.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:05.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:05.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:56:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:56:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:56:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:56:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:56:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:05.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:05.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:56:05.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:56:05.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:05.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:56:05.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:05.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:56:05.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:05.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:05.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:05.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:05.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:05.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:05.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:05.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:06.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:56:06.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:06.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:06.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:06.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:06.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:06.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:06.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:06.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:06.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:56:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:56:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:03 2026-02-22T19:56:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:56:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:56:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:02' 2026-02-22T19:56:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:56:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:56:06.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:06.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:56:06.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:06.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:06.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:06.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:06.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:06.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:06.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:03' 2026-02-22T19:56:06.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:56:03' 2026-02-22T19:56:06.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:06.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:06.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:06.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:06.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:03' 2026-02-22T19:56:06.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:06.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:06.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:06.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 19:56:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:56:02' 2026-02-22T19:56:06.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:06.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:03' 2026-02-22T19:56:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:06.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:56:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:06.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:56:06.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:56:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:06.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:07.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:56:07.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:07.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:07.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:07.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:07.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:07.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:07.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:07.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:07.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:02 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:03' 2026-02-22T19:56:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:56:07.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:56:07.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:07.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:56:07.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:07.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:07.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:07.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:07.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:07.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:07.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:02' 2026-02-22T19:56:07.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:56:02' 2026-02-22T19:56:07.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:07.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:07.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:07.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:07.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:02' 2026-02-22T19:56:07.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:07.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:07.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:07.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:56:02cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 19:56:03' 2026-02-22T19:56:07.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:07.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:07.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:02' 2026-02-22T19:56:07.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:07.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:56:07.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:07.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:56:07.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:07.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:07.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:07.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T19:56:07.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:07.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:07.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:07.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:56:07.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T19:56:08.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:08.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:09.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:56:09.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:09.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:09.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:09.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:56:09.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:09.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:56:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:56:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:56:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:56:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:09.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:56:09.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:09.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:09.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:56:09.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:56:09.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:09.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:09.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:56:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:56:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:56:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:56:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:09.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:56:09.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:56:09.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:56:09.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:09.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:56:09.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:56:09.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:09.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:09.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:09.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:09.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:09.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:56:09.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:56:09.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:09.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:09.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:56:09.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:09.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:09.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:09.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:09.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:56:09.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:56:09.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:09.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:56:09.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:56:09.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:56:09.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:09.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T19:56:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:56:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:56:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:56:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T19:56:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T19:56:13.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 29 2026-02-22T19:56:13.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T19:56:13.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:56:13.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:13.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:13.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:13.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:56:13.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:56:13.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:13.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:13.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:13.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:56:13.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:13.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:13.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:13.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:13.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:13.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:13.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:13.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:56:13.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:56:13.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:56:13.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=29 2026-02-22T19:56:13.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:56:13.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:56:13.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:56:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:56:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:56:14.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:56:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=29 2026-02-22T19:56:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 29 = 29 2026-02-22T19:56:14.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:56:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:14.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:15.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:56:15.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:15.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:15.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:15.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:15.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:15.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:15.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:15.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:15.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:56:15.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:15.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:15.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:56:15.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:56:15.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:15.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:13 2026-02-22T19:56:15.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:15.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:56:15.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:56:15.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790167,"replay_state":"idle"} 2026-02-22T19:56:15.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:07 2026-02-22T19:56:15.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:56:15.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 29 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.d03f9fae-5e2b-4fb3-a4a4-d8319d3997ee (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T19:56:15.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:56:15.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:56:15.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:56:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:15.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:56:15.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:15.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:56:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:15.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:15.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:15.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:13' 2026-02-22T19:56:15.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:56:13' 2026-02-22T19:56:15.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:15.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:15.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:15.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:15.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:13' 2026-02-22T19:56:15.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:15.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:15.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:15.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 19:56:13cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790167,"replay_state":"idle"}2026-02-22 19:56:07' 2026-02-22T19:56:15.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:15.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:15.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:13' 2026-02-22T19:56:15.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:15.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:56:15.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:56:15.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:56:15.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:56:15.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:15.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:15.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T19:56:15.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:15.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:15.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:15.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:15.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:15.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:15.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:15.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T19:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:16.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:16.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:56:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:56:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790167,"replay_state":"idle"} 2026-02-22T19:56:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:07 2026-02-22T19:56:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:56:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:56:16.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:56:16.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:13' 2026-02-22T19:56:16.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:56:16.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:56:16.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790167,"replay_state":"idle"} 2026-02-22T19:56:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:56:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:56:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:16.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:07' 2026-02-22T19:56:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 19:56:07' 2026-02-22T19:56:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:07' 2026-02-22T19:56:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:16.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:16.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790167,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 19:56:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 19:56:13' 2026-02-22T19:56:16.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:16.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:16.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:07' 2026-02-22T19:56:16.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:16.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:56:16.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:56:16.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:56:16.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:56:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T19:56:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T19:56:16.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:56:16.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T19:56:16.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:16.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:16.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:16.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T19:56:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:56:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:56:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:17.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:56:17.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:17.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:17.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:17.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:56:17.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:17.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:17.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:56:17.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:17.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:56:17.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:17.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:56:17.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:56:17.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:17.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:17.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:17.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:56:17.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:17.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:17.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:56:17.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:56:17.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:17.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:17.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:56:17.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:56:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:17.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:56:17.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:17.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:17.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:17.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:17.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:17.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:56:17.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:56:17.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:17.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:56:17.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:56:17.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:56:17.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:17.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:56:19.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:56:19.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:19.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:19.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:19.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:56:19.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:19.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:19.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:19.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:56:19.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:56:19.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:19.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:56:19.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:56:19.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:19.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:19.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:56:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:56:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:56:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:56:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:56:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:56:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:19.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:19.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:56:19.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:19.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:19.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:19.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:56:19.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:56:19.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:56:19.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:19.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:56:19.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:56:19.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:19.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:19.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:19.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:19.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:19.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:56:19.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:56:19.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:19.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:19.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:56:19.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:19.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:19.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:19.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:19.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:19.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:56:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:56:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:19.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:56:19.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:19.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:56:19.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:19.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:19.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:19.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:19.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:19.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:19.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:19.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:20.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:56:20.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:20.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:20.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:20.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:20.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:20.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:20.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:20.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:20.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:56:20.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:20.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:20.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:20.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:56:20.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:20.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:18 2026-02-22T19:56:20.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:20.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:56:20.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:20.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:56:20.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:18' 2026-02-22T19:56:20.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:56:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:56:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:20.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:56:20.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:20.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:20.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:20.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:18' 2026-02-22T19:56:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:56:18' 2026-02-22T19:56:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:18' 2026-02-22T19:56:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:56:18cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 19:56:18' 2026-02-22T19:56:21.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:21.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:21.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:18' 2026-02-22T19:56:21.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:21.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:56:21.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:21.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:56:21.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:21.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:21.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:21.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:56:21.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:21.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:21.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:21.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:21.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:21.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:21.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:21.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:22.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:56:22.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:22.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:22.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:22.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:22.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:22.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:22.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:22.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:22.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:56:22.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:22.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:22.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:22.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:56:22.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:22.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:18 2026-02-22T19:56:22.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:22.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:56:22.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:22.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:56:22.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:18' 2026-02-22T19:56:22.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:56:22.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:56:22.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:22.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:56:22.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:22.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:22.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:22.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:22.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:22.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:22.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:22.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:22.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:18' 2026-02-22T19:56:22.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:56:18' 2026-02-22T19:56:22.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:22.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:22.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:22.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:22.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:18' 2026-02-22T19:56:22.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:22.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:22.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 19:56:18cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:56:18' 2026-02-22T19:56:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:22.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:18' 2026-02-22T19:56:22.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:22.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:56:22.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:22.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:56:22.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:22.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T19:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:56:22.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T19:56:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:56:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:56:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:24.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:56:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:56:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:56:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:56:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:56:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:25.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:56:25.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:56:25.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:25.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:25.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:56:25.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:56:25.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:25.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:25.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:25.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:25.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:56:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:56:25.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:25.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:25.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:25.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:25.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:25.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:56:25.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:56:25.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:25.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:56:25.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:56:25.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:56:25.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:25.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:56:25.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:25.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:56:29.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 32 2026-02-22T19:56:29.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:56:29.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:56:29.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:29.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:29.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:29.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:29.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:56:29.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:29.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:29.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:56:29.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:56:29.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:29.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:56:29.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:56:29.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:56:29.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=32 2026-02-22T19:56:29.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:56:29.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:56:29.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:56:29.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:29.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:56:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:56:29.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:56:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=32 2026-02-22T19:56:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 32 = 32 2026-02-22T19:56:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T19:56:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:29.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:29.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:29.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:29.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T19:56:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:30.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:30.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:30.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:30.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:30.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:30.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:30.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:30.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:56:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:56:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:56:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:27 2026-02-22T19:56:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:30.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:56:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:56:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790183,"replay_state":"idle"} 2026-02-22T19:56:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:23 2026-02-22T19:56:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:56:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 32 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.77afa438-f873-48f4-ae54-afcbbdeced18 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:56:30.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:56:30.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:56:30.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:30.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:56:30.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:30.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:56:30.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:30.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:56:30.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:30.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:30.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:30.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:30.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:27' 2026-02-22T19:56:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 19:56:27' 2026-02-22T19:56:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:27' 2026-02-22T19:56:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:30.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 19:56:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790183,"replay_state":"idle"}2026-02-22 19:56:23' 2026-02-22T19:56:30.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:30.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:30.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:27' 2026-02-22T19:56:30.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:30.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:56:30.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:56:30.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:56:30.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:56:30.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:30.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:30.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:56:30.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:30.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:30.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:30.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:30.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:30.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:30.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:30.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:31.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:56:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:56:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790183,"replay_state":"idle"} 2026-02-22T19:56:31.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:23 2026-02-22T19:56:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:56:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:56:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:56:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:27' 2026-02-22T19:56:31.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:56:31.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:56:31.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:31.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790183,"replay_state":"idle"} 2026-02-22T19:56:31.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:31.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:56:31.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:31.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:56:31.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:31.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:31.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:31.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:23' 2026-02-22T19:56:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:56:23' 2026-02-22T19:56:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:23' 2026-02-22T19:56:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:31.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:31.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:31.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790183,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:56:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:56:27' 2026-02-22T19:56:31.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:31.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:31.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:23' 2026-02-22T19:56:31.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:31.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:56:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:56:31.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:56:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:56:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T19:56:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T19:56:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:31.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T19:56:32.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:56:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:56:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:56:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:32.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:32.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:56:32.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:56:32.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:32.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:56:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:56:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:56:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:56:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:56:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:56:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:33.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:33.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:56:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:56:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:33.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:56:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:56:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:56:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:33.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:33.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:33.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:33.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:56:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:56:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:56:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:56:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:56:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:56:35.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:56:35.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:35.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:35.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:35.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:56:35.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:35.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:35.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:56:35.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:35.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:56:35.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:35.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:56:35.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:56:35.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:35.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:35.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:35.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:35.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:56:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:56:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:56:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:56:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:56:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:35.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:56:35.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:56:35.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:35.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:35.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:35.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:35.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:35.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:35.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:35.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:35.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:56:35.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:56:35.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:56:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:56:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:35.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:35.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:36.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:56:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:36.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:56:36.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:56:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:33 2026-02-22T19:56:36.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:36.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:56:36.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:36.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:56:36.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:32' 2026-02-22T19:56:36.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:56:36.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:56:36.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:36.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:56:36.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:36.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:36.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:36.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:36.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:36.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:36.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:36.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:36.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:33' 2026-02-22T19:56:36.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:56:33' 2026-02-22T19:56:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:33' 2026-02-22T19:56:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 19:56:33cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:56:32' 2026-02-22T19:56:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:36.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:33' 2026-02-22T19:56:36.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:36.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:56:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:56:36.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:36.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:36.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:36.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:56:36.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:36.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:36.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:36.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:36.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:36.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:37.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:56:37.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:37.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:37.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:37.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:37.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:37.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:37.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:37.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:37.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:56:37.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:37.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:37.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:37.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:56:37.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:37.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:32 2026-02-22T19:56:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:56:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:56:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:33' 2026-02-22T19:56:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:56:37.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:56:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:56:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:37.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:37.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:37.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:32' 2026-02-22T19:56:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:56:32' 2026-02-22T19:56:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:32' 2026-02-22T19:56:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:37.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:56:32cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 19:56:33' 2026-02-22T19:56:37.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:37.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:32' 2026-02-22T19:56:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:37.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:56:37.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:37.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:56:37.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:37.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T19:56:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:56:37.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T19:56:38.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:56:38.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T19:56:38.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:38.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:38.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:38.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T19:56:38.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:38.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:38.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:38.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:56:38.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:56:38.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:38.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:56:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:39.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:56:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:39.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:56:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:56:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:56:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:56:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:39.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:39.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:56:39.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:39.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:56:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:56:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:56:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:56:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:56:39.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:39.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:39.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:39.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:39.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:39.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:56:39.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:56:39.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:39.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:39.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:56:39.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:39.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:39.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:39.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:39.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:39.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:56:39.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:56:39.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:39.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:56:39.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:56:39.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:56:39.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:39.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T19:56:39.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:56:39.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:39.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:39.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:39.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:56:39.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:56:39.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T19:56:39.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:39.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:39.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:39.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T19:56:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 35 2026-02-22T19:56:43.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T19:56:43.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:56:43.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:43.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:43.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:43.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:56:43.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:56:43.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:43.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:43.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:43.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:56:43.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:43.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:43.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:43.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:43.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:43.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:43.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:43.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:56:43.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:56:43.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:56:43.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=35 2026-02-22T19:56:43.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:56:43.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:56:43.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:56:43.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:44.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:56:44.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:56:44.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:56:44.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=35 2026-02-22T19:56:44.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 35 = 35 2026-02-22T19:56:44.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:44.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:56:44.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:44.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:44.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:44.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:44.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:44.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:44.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:44.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:45.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:56:45.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:45.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:45.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:45.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:45.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:45.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:45.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:45.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:45.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:56:45.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:45.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:45.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:56:45.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:56:45.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:45.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:43 2026-02-22T19:56:45.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:45.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:56:45.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:56:45.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790197,"replay_state":"idle"} 2026-02-22T19:56:45.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:37 2026-02-22T19:56:45.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:56:45.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 35 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.e0b09c28-d5a5-4409-9e0a-d6d234222183 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T19:56:45.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:56:45.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:56:45.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:45.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:56:45.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:45.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:56:45.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:45.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:56:45.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:45.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:45.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:45.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:43' 2026-02-22T19:56:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:56:43' 2026-02-22T19:56:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:43' 2026-02-22T19:56:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:45.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 19:56:43cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790197,"replay_state":"idle"}2026-02-22 19:56:37' 2026-02-22T19:56:45.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:45.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:45.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:43' 2026-02-22T19:56:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:45.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:56:45.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:56:45.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:56:45.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:56:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:45.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T19:56:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:45.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:46.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T19:56:46.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:46.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:46.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:46.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:46.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:46.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:46.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:46.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:46.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:56:46.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:46.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:46.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:56:46.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790197,"replay_state":"idle"} 2026-02-22T19:56:46.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:46.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:37 2026-02-22T19:56:46.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:46.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:56:46.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:56:46.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:56:46.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:43' 2026-02-22T19:56:46.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:56:46.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:56:46.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:46.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790197,"replay_state":"idle"} 2026-02-22T19:56:46.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:46.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:56:46.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:46.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:56:46.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:46.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:46.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:46.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:37' 2026-02-22T19:56:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 19:56:37' 2026-02-22T19:56:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:56:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:37' 2026-02-22T19:56:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:46.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790197,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 19:56:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 19:56:43' 2026-02-22T19:56:46.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:46.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:37' 2026-02-22T19:56:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:46.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:56:46.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:56:46.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:56:46.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:56:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T19:56:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T19:56:46.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:56:46.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T19:56:46.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:46.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:46.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T19:56:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:56:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:56:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:46.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:47.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:56:47.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:47.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:47.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:47.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:56:47.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:47.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:47.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:56:47.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:47.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:56:47.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:47.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:56:47.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:56:47.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:47.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:47.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:47.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:56:47.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:47.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:47.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:56:47.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:47.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:47.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:47.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:47.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:56:47.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:56:47.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:47.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:56:47.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:56:47.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:56:47.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:47.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:56:49.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:56:49.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:49.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:49.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:56:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:56:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:56:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:56:49.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:56:49.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:49.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:49.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:49.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:56:49.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:49.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:49.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:56:49.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:56:49.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:49.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:49.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:56:49.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:56:49.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:49.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:49.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:56:49.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:49.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:49.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:56:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:56:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:56:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:56:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:56:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:56:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:56:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:56:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:56:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:56:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:49.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:56:49.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:49.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:49.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:56:49.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:56:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:56:49.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:49.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:56:49.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:56:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:56:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:50.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:50.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:56:50.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:50.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:50.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:50.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:56:50.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:50.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:48 2026-02-22T19:56:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:56:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:56:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:48' 2026-02-22T19:56:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:56:50.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:56:50.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:50.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:56:50.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:50.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:50.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:50.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:50.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:50.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:50.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:50.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:50.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:48' 2026-02-22T19:56:50.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:56:48' 2026-02-22T19:56:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:48' 2026-02-22T19:56:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:50.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:56:48cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 19:56:48' 2026-02-22T19:56:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:51.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:51.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:48' 2026-02-22T19:56:51.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:51.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:56:51.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:51.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:56:51.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:56:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:51.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:51.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:51.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:51.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:52.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:56:52.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:52.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:52.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:52.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:52.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:52.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:52.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:52.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:52.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:56:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:56:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:56:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:56:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:56:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:48 2026-02-22T19:56:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:56:52.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:56:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:56:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:56:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:48' 2026-02-22T19:56:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:56:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:56:52.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:56:52.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:56:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:56:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:52.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:56:52.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:52.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:56:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:56:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:56:52.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:56:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:48' 2026-02-22T19:56:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:56:48' 2026-02-22T19:56:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:56:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:48' 2026-02-22T19:56:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:56:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:56:52.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:56:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 19:56:48cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:56:48' 2026-02-22T19:56:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:56:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:56:52.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:48' 2026-02-22T19:56:52.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:56:52.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:56:52.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:56:52.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:56:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:56:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T19:56:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:56:52.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T19:56:54.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:56:54.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:56:54.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:54.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:54.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:54.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:56:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:56:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:56:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:54.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:56:55.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:56:55.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:55.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:56:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:56:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:56:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:56:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:56:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:56:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:56:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:56:55.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:56:55.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:55.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:56:55.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:56:55.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:56:55.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:56:55.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:56:55.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:56:55.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:56:55.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:56:55.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:56:55.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:56:55.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:56:55.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:56:55.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:56:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:55.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:56:55.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:56:55.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:56:55.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:56:55.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:56:55.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:56:55.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:56:55.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:56:55.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:56:55.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:55.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:56:55.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:56:55.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:55.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:55.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:55.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:56:55.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:56:55.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:56:55.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:55.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:55.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:55.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:56:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 38 2026-02-22T19:56:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:56:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:56:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:56:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:56:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:59.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:59.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:56:59.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:56:59.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:56:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:56:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:56:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:56:59.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:56:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=38 2026-02-22T19:56:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:56:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:56:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:56:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:56:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:56:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:56:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:56:59.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:56:59.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=38 2026-02-22T19:56:59.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 38 = 38 2026-02-22T19:56:59.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:56:59.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T19:56:59.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:56:59.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:56:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:56:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:56:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:56:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:56:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:56:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T19:57:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:57:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:57:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:57:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:57 2026-02-22T19:57:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:57:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:57:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790213,"replay_state":"idle"} 2026-02-22T19:57:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:53 2026-02-22T19:57:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:57:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 38 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.656f29b3-9d5a-4126-9c45-da43b8a27b18 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:57:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:57:00.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:57:00.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:00.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:57:00.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:00.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:57:00.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:00.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:57:00.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:00.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:00.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:00.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:57' 2026-02-22T19:57:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 19:56:57' 2026-02-22T19:57:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:00.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:00.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:00.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:57' 2026-02-22T19:57:00.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:00.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:00.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:00.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 19:56:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790213,"replay_state":"idle"}2026-02-22 19:56:53' 2026-02-22T19:57:00.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:00.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:00.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:57' 2026-02-22T19:57:00.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:57:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:57:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:57:00.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:57:00.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:00.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:00.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:57:00.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:00.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:00.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:00.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:00.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:00.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:00.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:00.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:01.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:57:01.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:01.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:57:01.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:01.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:57:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790213,"replay_state":"idle"} 2026-02-22T19:57:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:53 2026-02-22T19:57:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:57:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:57:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:57:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:56:57' 2026-02-22T19:57:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:57:01.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:57:01.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790213,"replay_state":"idle"} 2026-02-22T19:57:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:57:01.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:01.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:57:01.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:01.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:01.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:01.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:53' 2026-02-22T19:57:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:56:53' 2026-02-22T19:57:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:01.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:56:53' 2026-02-22T19:57:01.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:01.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:01.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:01.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790213,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:56:53cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:56:57' 2026-02-22T19:57:01.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:01.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:01.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:56:53' 2026-02-22T19:57:01.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:01.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:57:01.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:57:01.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:57:01.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:57:01.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:01.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:01.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T19:57:01.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T19:57:01.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:01.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:01.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:01.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T19:57:02.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:57:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:57:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:57:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:57:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:57:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:57:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:57:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:57:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:57:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:57:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:57:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:03.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:03.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:57:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:57:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:03.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:57:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:57:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:57:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:03.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:57:03.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:57:03.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:03.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:03.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:03.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:03.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:57:03.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:57:03.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:03.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:57:03.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:57:03.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:57:03.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:03.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:57:05.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:57:05.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:57:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:57:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:57:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:57:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:57:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:05.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:05.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:05.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:57:05.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:57:05.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:05.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:05.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:05.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:57:05.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:57:05.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:05.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:05.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:05.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:05.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:57:05.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:57:05.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:05.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:57:05.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:05.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:57:05.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:05.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:05.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:05.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:05.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:05.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:05.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:05.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:57:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:03 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:02' 2026-02-22T19:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:57:06.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:57:06.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:57:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:06.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:06.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:06.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:06.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:06.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:03' 2026-02-22T19:57:06.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:57:03' 2026-02-22T19:57:06.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:06.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:06.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:06.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:06.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:03' 2026-02-22T19:57:06.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:06.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:06.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 19:57:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:57:02' 2026-02-22T19:57:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:06.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:03' 2026-02-22T19:57:06.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:06.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:57:06.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:06.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:57:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:06.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:57:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:07.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:57:07.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:07.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:07.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:07.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:07.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:07.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:07.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:07.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:07.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:57:07.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:07.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:07.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:07.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:07.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:07.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:02 2026-02-22T19:57:07.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:07.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:57:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:57:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:03' 2026-02-22T19:57:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:57:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:57:07.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:07.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:57:07.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:07.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:07.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:07.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:07.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:07.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:02' 2026-02-22T19:57:07.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:57:02' 2026-02-22T19:57:07.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:07.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:07.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:07.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:07.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:02' 2026-02-22T19:57:07.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:07.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:07.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:07.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:57:02cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 19:57:03' 2026-02-22T19:57:07.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:07.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:07.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:02' 2026-02-22T19:57:07.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:57:07.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:07.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:57:07.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:07.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:07.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:07.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T19:57:07.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:07.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:07.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:07.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:57:07.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T19:57:08.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:57:08.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T19:57:08.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:08.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:08.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:08.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T19:57:08.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:08.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:08.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:08.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:57:08.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:57:08.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:08.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:57:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:57:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:09.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:09.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:57:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:57:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:57:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:57:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:09.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:09.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:57:09.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:09.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:09.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:57:09.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:57:09.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:09.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:09.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:57:09.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:57:09.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:09.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:09.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:57:09.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:09.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:09.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:57:09.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:09.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:09.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:09.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:57:09.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:57:09.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:09.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:09.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:57:09.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:09.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:09.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:09.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:09.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:09.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:57:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:57:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:09.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:57:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:57:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:57:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T19:57:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:57:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:57:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:57:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T19:57:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T19:57:13.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 41 2026-02-22T19:57:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T19:57:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:57:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:13.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:13.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:57:13.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:57:13.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:13.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:13.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:13.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:57:13.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:13.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:13.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:13.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:13.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:13.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:13.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:13.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:57:13.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:57:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:57:13.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=41 2026-02-22T19:57:13.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:57:13.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:13.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:57:13.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:13.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:13.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:13.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:13.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:13.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:57:13.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:57:13.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:57:13.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=41 2026-02-22T19:57:13.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 41 = 41 2026-02-22T19:57:13.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:13.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:57:13.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:13.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:13.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:14.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:57:14.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:14.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:14.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:14.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:14.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:14.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:14.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:14.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:14.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:57:14.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:57:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:57:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:13 2026-02-22T19:57:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:57:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:57:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790227,"replay_state":"idle"} 2026-02-22T19:57:14.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:07 2026-02-22T19:57:14.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:57:14.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 41 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.e99499df-6553-4023-9271-6bea768b2cb4 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T19:57:14.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:57:14.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:57:14.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:14.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:57:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:57:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:57:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:14.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:14.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:14.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:14.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:13' 2026-02-22T19:57:14.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:57:13' 2026-02-22T19:57:14.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:14.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:14.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:14.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:14.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:13' 2026-02-22T19:57:14.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:14.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:14.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:14.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 19:57:13cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790227,"replay_state":"idle"}2026-02-22 19:57:07' 2026-02-22T19:57:14.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:14.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:14.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:13' 2026-02-22T19:57:14.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:14.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:57:14.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:57:14.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:57:14.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:57:14.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:14.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:14.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T19:57:14.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:14.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:14.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:14.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:14.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:14.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:14.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:14.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T19:57:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:15.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:15.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:57:15.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:15.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:15.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:57:15.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790227,"replay_state":"idle"} 2026-02-22T19:57:15.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:15.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:07 2026-02-22T19:57:15.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:15.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:57:15.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:57:15.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:57:15.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:13' 2026-02-22T19:57:15.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:57:15.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:57:15.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:15.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790227,"replay_state":"idle"} 2026-02-22T19:57:15.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:15.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:57:15.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:15.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:57:15.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:15.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:15.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:07' 2026-02-22T19:57:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 19:57:07' 2026-02-22T19:57:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:07' 2026-02-22T19:57:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:15.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:15.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790227,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 19:57:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 19:57:13' 2026-02-22T19:57:15.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:15.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:15.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:07' 2026-02-22T19:57:15.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:57:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:57:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:57:15.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:57:15.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:15.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:15.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T19:57:15.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:15.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:15.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:15.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T19:57:16.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:57:16.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T19:57:16.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:16.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:16.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:16.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T19:57:16.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:16.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:16.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:16.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:57:16.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:57:16.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:16.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:57:17.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:17.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:17.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:57:17.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:17.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:57:17.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:17.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:57:17.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:57:17.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:17.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:17.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:17.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:57:17.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:17.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:17.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:17.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:17.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:17.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:57:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:57:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:57:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:57:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:57:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:17.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:57:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:57:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:17.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:57:17.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:17.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:17.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:17.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:17.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:17.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:57:17.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:57:17.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:17.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:57:17.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:57:17.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:57:17.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:17.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:57:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:57:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:57:19.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:19.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:57:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:57:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:57:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:57:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:19.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:19.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:57:19.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:19.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:57:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:57:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:57:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:57:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:57:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:19.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:19.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:57:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:57:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:57:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:19.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:19.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:19.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:19.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:19.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:57:19.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:57:19.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:57:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:57:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:20.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:57:20.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:20.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:20.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:20.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:20.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:20.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:20.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:20.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:20.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:57:20.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:20.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:20.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:20.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:20.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:20.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:18 2026-02-22T19:57:20.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:20.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:57:20.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:20.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:57:20.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:18' 2026-02-22T19:57:20.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:57:20.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:57:20.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:57:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:20.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:20.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:20.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:20.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:20.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:18' 2026-02-22T19:57:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:57:18' 2026-02-22T19:57:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:18' 2026-02-22T19:57:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:20.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:20.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:20.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:57:18cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 19:57:18' 2026-02-22T19:57:20.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:20.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:20.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:18' 2026-02-22T19:57:20.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:20.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:57:20.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:20.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:57:20.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:57:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:20.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:57:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:21.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:21.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:57:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:57:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:18 2026-02-22T19:57:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:57:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:18' 2026-02-22T19:57:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:57:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:57:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:21.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:57:21.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:21.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:21.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:21.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:21.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:21.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:21.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:21.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:18' 2026-02-22T19:57:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:57:18' 2026-02-22T19:57:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:21.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:21.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:21.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:21.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:18' 2026-02-22T19:57:21.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:21.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:21.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:21.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 19:57:18cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:57:18' 2026-02-22T19:57:21.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:21.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:21.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:18' 2026-02-22T19:57:21.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:21.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:57:21.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:21.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:57:21.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:21.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:21.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:21.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T19:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:57:21.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T19:57:23.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:57:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:57:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:57:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:57:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:57:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:23.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:57:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:24.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:57:24.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:24.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:24.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:57:24.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:24.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:57:24.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:24.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:57:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:57:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:57:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:57:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:57:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:57:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:57:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:57:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:24.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:24.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:57:24.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:57:24.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:24.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:57:24.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:57:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:57:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:57:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:57:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:57:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:57:24.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:57:24.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:24.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:24.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:24.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:57:28.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 44 2026-02-22T19:57:28.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:57:28.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:57:28.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:28.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:28.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:28.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:28.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:57:28.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:28.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:28.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:57:28.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:57:28.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:57:28.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:57:28.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:57:28.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=44 2026-02-22T19:57:28.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:57:28.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:28.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:57:28.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:28.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:28.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:28.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:28.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:28.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:57:28.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:57:28.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=44 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 44 = 44 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:28.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T19:57:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:57:29.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:29.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:29.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:57:29.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:57:29.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:29.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:27 2026-02-22T19:57:29.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:29.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:57:29.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:57:29.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790242,"replay_state":"idle"} 2026-02-22T19:57:29.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:23 2026-02-22T19:57:29.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:57:29.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 44 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.dc775ae7-94f2-471b-a376-3787100a8af6 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:57:29.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:57:29.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:57:29.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:29.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:57:29.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:29.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:57:29.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:29.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:57:29.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:29.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:29.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:29.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:29.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:27' 2026-02-22T19:57:29.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 19:57:27' 2026-02-22T19:57:29.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:29.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:29.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:29.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:29.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:27' 2026-02-22T19:57:29.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:29.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:29.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:29.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 19:57:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790242,"replay_state":"idle"}2026-02-22 19:57:23' 2026-02-22T19:57:29.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:29.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:29.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:27' 2026-02-22T19:57:29.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:57:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:57:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:57:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:57:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:57:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:29.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:57:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:30.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:30.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:30.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:57:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:57:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790242,"replay_state":"idle"} 2026-02-22T19:57:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:23 2026-02-22T19:57:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:57:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:57:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:57:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:27' 2026-02-22T19:57:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:57:30.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:57:30.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:30.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790242,"replay_state":"idle"} 2026-02-22T19:57:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:57:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:57:30.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:30.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:30.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:30.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:30.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:23' 2026-02-22T19:57:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:57:23' 2026-02-22T19:57:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:23' 2026-02-22T19:57:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:30.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790242,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:57:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:57:27' 2026-02-22T19:57:30.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:30.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:30.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:23' 2026-02-22T19:57:30.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:30.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:57:30.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:57:30.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:57:30.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:57:30.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:30.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:30.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T19:57:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T19:57:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T19:57:31.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:57:31.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:57:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:57:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:57:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:57:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:31.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:32.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:57:32.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:32.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:32.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:32.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:57:32.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:32.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:32.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:57:32.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:57:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:57:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:57:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:57:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:32.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:32.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:32.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:57:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:57:32.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:57:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:57:32.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:57:32.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:32.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:57:34.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:57:34.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:34.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:34.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:57:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:34.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:34.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:34.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:57:34.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:57:34.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:57:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:57:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:34.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:57:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:33 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:32' 2026-02-22T19:57:35.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:57:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:57:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:57:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:35.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:35.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:35.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:35.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:35.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:35.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:33' 2026-02-22T19:57:35.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:57:33' 2026-02-22T19:57:35.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:35.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:35.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:35.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:35.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:33' 2026-02-22T19:57:35.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:35.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:35.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:35.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:57:33cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:57:32' 2026-02-22T19:57:35.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:35.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:35.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:33' 2026-02-22T19:57:35.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:35.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:57:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:35.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:57:35.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:35.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:35.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:35.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:57:35.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:35.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:35.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:35.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:35.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:35.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:35.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:35.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:36.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:57:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:57:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:32 2026-02-22T19:57:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:57:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:33' 2026-02-22T19:57:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:57:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:57:36.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:36.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:57:36.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:36.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:36.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:36.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:36.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:36.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:36.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:36.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:32' 2026-02-22T19:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:57:32' 2026-02-22T19:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:32' 2026-02-22T19:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:57:32cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:57:33' 2026-02-22T19:57:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:32' 2026-02-22T19:57:36.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:36.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:57:36.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:36.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:57:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:36.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T19:57:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:57:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T19:57:37.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:57:37.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T19:57:37.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:37.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:37.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:37.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T19:57:37.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:37.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:37.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:37.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:57:37.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:57:37.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:37.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:38.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:57:38.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:38.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:38.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:38.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:57:38.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:38.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:38.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:57:38.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:38.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:57:38.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:38.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:57:38.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:57:38.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:38.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:38.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:57:38.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:38.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:38.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:57:38.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:57:38.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:38.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:38.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:57:38.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:57:38.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:38.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:38.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:57:38.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:38.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:38.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:57:38.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:57:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:57:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:57:38.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:57:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:57:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:38.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:38.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:38.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:57:38.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:57:38.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:38.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:38.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:38.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:38.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:57:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:57:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:57:38.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:57:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:57:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T19:57:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:57:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:57:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:57:38.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T19:57:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T19:57:42.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 47 2026-02-22T19:57:42.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T19:57:42.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:57:42.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:42.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:42.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:42.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:57:42.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:57:42.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:42.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:57:42.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:42.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:42.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:42.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:42.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:42.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:42.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:42.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:57:42.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:57:42.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:57:42.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=47 2026-02-22T19:57:42.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:57:42.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:42.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:57:42.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:42.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:42.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:42.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:42.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:42.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:57:42.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:57:42.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:57:42.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=47 2026-02-22T19:57:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 47 = 47 2026-02-22T19:57:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:57:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:42.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:57:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:43.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:57:43.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:43 2026-02-22T19:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790256,"replay_state":"idle"} 2026-02-22T19:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:37 2026-02-22T19:57:43.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:57:43.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 47 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.1b1287fe-41de-4771-89d4-91d118293321 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T19:57:43.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:57:43.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:57:43.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:43.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:57:43.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:43.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:57:43.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:43.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:57:43.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:43.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:43.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:43.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:43.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:43' 2026-02-22T19:57:43.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:57:43' 2026-02-22T19:57:43.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:43.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:43.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:43.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:43.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:43' 2026-02-22T19:57:43.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:43.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:43.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:43.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 19:57:43cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790256,"replay_state":"idle"}2026-02-22 19:57:37' 2026-02-22T19:57:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:43.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:43' 2026-02-22T19:57:43.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:43.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:57:43.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:57:43.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:57:43.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:57:43.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:43.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:43.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T19:57:43.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:43.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:43.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:43.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:43.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:43.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:43.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:43.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T19:57:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:44.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790256,"replay_state":"idle"} 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:37 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:43' 2026-02-22T19:57:44.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:57:44.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:57:44.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:44.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790256,"replay_state":"idle"} 2026-02-22T19:57:44.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:44.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:57:44.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:44.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:57:44.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:44.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:44.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:44.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:37' 2026-02-22T19:57:44.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 19:57:37' 2026-02-22T19:57:44.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:44.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:44.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:44.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:57:44.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:37' 2026-02-22T19:57:44.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:44.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:44.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:45.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790256,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 19:57:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 19:57:43' 2026-02-22T19:57:45.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:45.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:45.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:37' 2026-02-22T19:57:45.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:57:45.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:57:45.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:57:45.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:57:45.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:45.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:45.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T19:57:45.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T19:57:45.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:57:45.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T19:57:45.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:45.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:45.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:45.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T19:57:45.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:57:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:57:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:46.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:57:46.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:46.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:46.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:46.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:57:46.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:46.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:46.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:57:46.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:46.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:57:46.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:46.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:57:46.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:57:46.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:46.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:46.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:46.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:57:46.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:46.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:46.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:57:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:46.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:46.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:57:46.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:57:46.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:46.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:57:46.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:57:46.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:57:46.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:46.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:57:48.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:57:48.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:48.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:48.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:48.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:57:48.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:48.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:57:48.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:57:48.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:48.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:57:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:57:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:57:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:48.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:57:48.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:48.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:48.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:57:48.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:48.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:48.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:57:48.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:57:48.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:57:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:57:48.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:48.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:49.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:57:49.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:49.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:49.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:49.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:49.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:49.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:49.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:49.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:49.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:57:49.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:49.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:49.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:49.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:48 2026-02-22T19:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:47' 2026-02-22T19:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:57:49.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:57:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:49.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:49.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:49.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:49.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:49.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:49.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:49.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:48' 2026-02-22T19:57:49.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:57:48' 2026-02-22T19:57:49.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:49.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:49.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:49.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:49.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:48' 2026-02-22T19:57:49.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:49.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:49.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:57:48cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:57:47' 2026-02-22T19:57:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:49.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:49.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:48' 2026-02-22T19:57:49.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:57:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:49.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:57:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:49.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:49.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:49.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:57:49.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:49.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:49.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:49.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:49.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:49.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:50.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:57:50.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:50.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:50.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:50.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:50.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:50.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:50.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:50.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:50.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:57:50.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:57:50.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:57:50.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:50.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:50.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:57:50.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:47 2026-02-22T19:57:50.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:57:50.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:57:50.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:57:50.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:57:50.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:48' 2026-02-22T19:57:50.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:57:50.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:57:50.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:57:50.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:57:50.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:57:50.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:50.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:57:50.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:50.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:57:50.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:57:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:57:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:57:50.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:47' 2026-02-22T19:57:50.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:57:47' 2026-02-22T19:57:50.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:50.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:50.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:50.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:57:50.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:47' 2026-02-22T19:57:50.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:57:50.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:57:50.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:57:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:57:47cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:57:48' 2026-02-22T19:57:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:57:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:57:50.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:47' 2026-02-22T19:57:50.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:57:50.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:57:50.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:57:50.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:57:50.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:57:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T19:57:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:57:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T19:57:52.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:52.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:57:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:57:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:57:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:53.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:53.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:57:53.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:53.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:57:53.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:53.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:57:53.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:57:53.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:53.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:53.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:53.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:53.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:53.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:53.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:57:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:57:53.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T19:57:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:53.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:57:55.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:57:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:57:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:57:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:57:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:57:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:57:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:57:55.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:55.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:57:55.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:57:55.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:57:55.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:55.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:57:55.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:57:55.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:57:55.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:57:55.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:57:55.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:57:55.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:57:55.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:57:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:57:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:57:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:57:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:57:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:57:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:57:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:57:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:57:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:57:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:57:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:57:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:57:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:57:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:55.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:55.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:57:55.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:57:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:57:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:57:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:57:55.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:57:55.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:57:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:57:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:57:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:57:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:57:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:57:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:57:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:57:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:57:59.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 50 2026-02-22T19:57:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:57:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:57:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:57:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:57:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:59.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:59.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:57:59.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:57:59.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:57:59.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:59.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:59.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:59.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:59.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:59.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:57:59.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:57:59.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:57:59.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:57:59.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=50 2026-02-22T19:57:59.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:57:59.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:57:59.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:57:59.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:59.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:57:59.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:59.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:59.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:57:59.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:57:59.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:57:59.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=50 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 50 = 50 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:57:59.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T19:58:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:58:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:58:00.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:58:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:57 2026-02-22T19:58:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:58:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:58:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790271,"replay_state":"idle"} 2026-02-22T19:58:00.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:53 2026-02-22T19:58:00.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:58:00.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 50 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.e064bcdf-eb72-404d-86a8-70fccbac3866 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:58:00.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:58:00.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:58:00.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:00.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:58:00.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:00.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:58:00.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:00.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:58:00.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:00.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:00.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:00.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:00.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:57' 2026-02-22T19:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 19:57:57' 2026-02-22T19:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:57' 2026-02-22T19:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:00.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:00.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 19:57:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790271,"replay_state":"idle"}2026-02-22 19:57:53' 2026-02-22T19:58:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:00.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:57' 2026-02-22T19:58:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:00.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:58:00.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:58:00.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:58:00.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:58:00.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:00.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:00.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:58:00.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:01.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:58:01.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:58:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:58:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790271,"replay_state":"idle"} 2026-02-22T19:58:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:53 2026-02-22T19:58:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:58:01.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:58:01.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:58:01.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:57:57' 2026-02-22T19:58:01.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:58:01.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:58:01.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:01.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790271,"replay_state":"idle"} 2026-02-22T19:58:01.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:01.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:58:01.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:58:01.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:01.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:01.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:53' 2026-02-22T19:58:01.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:57:53' 2026-02-22T19:58:01.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:01.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:57:53' 2026-02-22T19:58:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:01.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790271,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:57:53cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:57:57' 2026-02-22T19:58:01.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:01.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:57:53' 2026-02-22T19:58:01.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:01.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:58:01.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:58:01.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:58:01.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:58:01.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:01.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:01.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T19:58:01.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T19:58:01.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:01.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:01.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:01.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T19:58:02.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:02.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:03.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:58:03.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:03.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:03.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:03.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:58:03.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:03.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:58:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:58:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:58:03.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:58:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:03.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:58:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:58:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:58:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:58:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:03.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:03.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:03.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:03.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:03.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:03.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:58:03.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:58:03.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:03.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:58:03.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:58:03.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:58:03.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:03.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:58:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:58:05.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:58:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:58:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:58:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:58:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:58:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:05.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:05.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:58:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:58:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:58:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:58:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:05.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:05.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:05.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:05.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:58:05.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:58:05.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:05.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:58:05.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:05.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:58:05.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:05.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:05.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:05.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:05.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:05.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:05.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:05.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:58:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:06.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:58:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:58:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:03 2026-02-22T19:58:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:58:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:02' 2026-02-22T19:58:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:58:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:58:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:58:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:06.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:06.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:03' 2026-02-22T19:58:06.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:58:03' 2026-02-22T19:58:06.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:06.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:06.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:06.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:06.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:03' 2026-02-22T19:58:06.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:06.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:06.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:06.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 19:58:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:58:02' 2026-02-22T19:58:06.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:06.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:03' 2026-02-22T19:58:06.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:06.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:58:06.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:06.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:58:06.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:06.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:06.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:06.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:58:06.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:06.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:06.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:06.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:06.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:06.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:06.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:06.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:58:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:07.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:58:07.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:07.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:07.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:07.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:02 2026-02-22T19:58:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:58:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:58:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:03' 2026-02-22T19:58:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:58:07.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:58:07.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:07.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:58:07.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:07.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:07.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:07.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:07.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:07.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:07.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:07.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:07.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:02' 2026-02-22T19:58:07.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:58:02' 2026-02-22T19:58:07.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:07.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:07.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:07.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:07.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:02' 2026-02-22T19:58:07.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:07.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:07.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:07.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:58:02cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 19:58:03' 2026-02-22T19:58:07.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:07.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:07.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:02' 2026-02-22T19:58:07.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:07.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:58:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:58:07.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:07.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:07.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:07.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T19:58:07.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:07.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:07.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:07.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:58:07.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T19:58:08.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:58:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T19:58:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T19:58:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:58:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:58:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:08.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:09.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:58:09.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:09.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:09.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:09.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:58:09.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:09.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:09.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:58:09.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:09.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:58:09.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:58:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:58:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:58:09.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:58:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:58:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:58:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:58:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:58:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:58:09.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:58:09.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:58:09.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:09.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:09.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:58:09.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:09.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:58:09.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:58:09.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:09.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:09.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:09.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T19:58:13.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 53 2026-02-22T19:58:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T19:58:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:58:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:58:13.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:58:13.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:13.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:13.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:13.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:58:13.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:13.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:13.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:13.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:13.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:13.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:13.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:13.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:58:13.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:58:13.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:58:13.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=53 2026-02-22T19:58:13.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:58:13.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:13.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:58:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:13.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:13.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:58:13.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:58:13.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=53 2026-02-22T19:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 53 = 53 2026-02-22T19:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:13.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:14.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:58:14.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:14.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:14.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:14.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:14.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:14.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:14.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:14.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:14.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:13 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790287,"replay_state":"idle"} 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:07 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 53 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.dbb51115-9f69-4460-a36c-acd8181458dd (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T19:58:15.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:58:15.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:58:15.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:15.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:58:15.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:15.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:58:15.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:15.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:58:15.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:15.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:15.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:15.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:15.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:13' 2026-02-22T19:58:15.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:58:13' 2026-02-22T19:58:15.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:15.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:15.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:15.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:15.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:13' 2026-02-22T19:58:15.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:15.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:15.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:15.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 19:58:13cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790287,"replay_state":"idle"}2026-02-22 19:58:07' 2026-02-22T19:58:15.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:15.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:15.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:13' 2026-02-22T19:58:15.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:15.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:58:15.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:58:15.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:15.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:16.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T19:58:16.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:16.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:16.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:16.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:16.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:16.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:16.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:16.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:16.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:58:16.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:16.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:16.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:58:16.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790287,"replay_state":"idle"} 2026-02-22T19:58:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:07 2026-02-22T19:58:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:58:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:58:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:58:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:13' 2026-02-22T19:58:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:58:16.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:58:16.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:16.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790287,"replay_state":"idle"} 2026-02-22T19:58:16.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:16.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:58:16.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:16.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:58:16.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:16.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:16.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:16.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:16.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:07' 2026-02-22T19:58:16.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 19:58:07' 2026-02-22T19:58:16.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:16.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:16.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:16.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:16.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:07' 2026-02-22T19:58:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:16.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790287,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 19:58:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 19:58:13' 2026-02-22T19:58:16.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:16.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:16.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:07' 2026-02-22T19:58:16.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:16.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:58:16.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:58:16.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:58:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:58:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T19:58:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:16.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T19:58:16.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:58:16.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T19:58:16.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T19:58:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:58:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:58:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:16.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:58:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:58:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:58:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:58:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:58:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:58:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:17.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:58:17.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:17.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:17.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:58:17.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:17.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:17.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:17.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:17.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:17.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:58:17.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:58:17.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:17.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:58:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:58:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:58:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:17.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:58:19.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:58:19.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:19.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:19.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:19.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:58:19.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:19.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:19.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:58:19.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:19.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:58:19.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:19.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:58:19.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:58:19.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:19.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:19.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:19.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:58:19.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:19.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:58:19.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:58:19.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:58:19.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:19.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:19.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:58:19.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:19.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:19.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:19.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:19.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:19.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:58:19.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:58:19.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:19.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:58:19.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:19.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:58:19.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:19.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:19.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:19.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:19.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:19.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:19.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:19.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:20.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:58:20.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:20.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:20.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:20.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:20.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:20.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:20.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:20.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:58:20.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:20.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:20.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:20.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:20.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:20.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:18 2026-02-22T19:58:20.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:20.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:58:20.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:20.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:58:20.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:18' 2026-02-22T19:58:20.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:58:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:58:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:20.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:58:20.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:20.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:20.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:20.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:20.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:20.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:20.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:20.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:20.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:18' 2026-02-22T19:58:20.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:58:18' 2026-02-22T19:58:20.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:18' 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:20.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:20.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:58:18cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 19:58:18' 2026-02-22T19:58:20.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:20.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:18' 2026-02-22T19:58:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:20.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:58:20.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:20.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:58:20.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:58:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:20.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:20.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:21.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:58:21.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:58:21.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:21.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:21.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:21.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:58:21.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:21.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:18 2026-02-22T19:58:21.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:58:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:18' 2026-02-22T19:58:21.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:58:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:58:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:21.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:58:21.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:21.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:21.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:21.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:21.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:21.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:21.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:21.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:21.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:18' 2026-02-22T19:58:21.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:58:18' 2026-02-22T19:58:21.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:21.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:21.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:21.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:21.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:18' 2026-02-22T19:58:21.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:21.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:21.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 19:58:18cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:58:18' 2026-02-22T19:58:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:21.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:18' 2026-02-22T19:58:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:21.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:58:21.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:21.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:58:21.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T19:58:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:58:21.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T19:58:23.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:58:23.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:58:23.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:23.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:23.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:23.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:58:23.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:23.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:23.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:23.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:58:23.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:58:23.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:23.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:24.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:58:24.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:24.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:24.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:24.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:24.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:24.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:24.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:58:24.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:58:24.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:24.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:24.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:24.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:58:24.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:24.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:24.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:58:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:58:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:58:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:58:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:58:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:58:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:58:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:24.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:24.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:24.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:58:24.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:58:24.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:58:24.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:58:24.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:24.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:24.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:24.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:58:28.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 56 2026-02-22T19:58:28.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:58:28.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:58:28.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:28.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:28.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:28.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:28.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:58:28.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:28.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:28.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:58:28.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:58:28.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:28.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:58:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:58:28.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:58:28.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=56 2026-02-22T19:58:28.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:58:28.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:28.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:58:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:28.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:28.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:58:28.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:58:28.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=56 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 56 = 56 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:28.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:29.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T19:58:29.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:29.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:29.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:29.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:29.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:29.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:29.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:29.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:29.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:58:29.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:29.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:27 2026-02-22T19:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790302,"replay_state":"idle"} 2026-02-22T19:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:23 2026-02-22T19:58:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:58:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 56 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.702c6eef-820e-462c-94d5-4cc1c3a8a9ed (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:58:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:58:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:58:29.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:29.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:58:29.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:29.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:58:29.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:29.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:58:29.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:29.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:29.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:27' 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 19:58:27' 2026-02-22T19:58:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:27' 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:29.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 19:58:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790302,"replay_state":"idle"}2026-02-22 19:58:23' 2026-02-22T19:58:29.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:29.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:29.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:27' 2026-02-22T19:58:29.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:58:29.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:58:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:58:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:29.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:29.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:29.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:29.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:29.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:30.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:58:30.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:30.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:30.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:30.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:30.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:30.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:30.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:30.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:30.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:58:30.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:30.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:30.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:58:30.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790302,"replay_state":"idle"} 2026-02-22T19:58:30.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:23 2026-02-22T19:58:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:58:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:58:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:58:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:27' 2026-02-22T19:58:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:58:30.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:58:30.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:30.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790302,"replay_state":"idle"} 2026-02-22T19:58:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:58:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:30.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:58:30.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:30.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:23' 2026-02-22T19:58:30.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:58:23' 2026-02-22T19:58:30.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:30.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:30.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:30.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:30.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:23' 2026-02-22T19:58:30.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:30.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:30.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:30.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790302,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:58:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:58:27' 2026-02-22T19:58:30.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:30.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:30.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:23' 2026-02-22T19:58:30.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:30.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:58:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:58:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:58:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:58:30.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:30.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:30.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T19:58:30.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T19:58:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:30.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T19:58:31.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:58:31.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:58:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:58:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:58:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:58:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:31.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:32.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:58:32.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:32.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:32.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:32.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:58:32.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:32.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:32.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:58:32.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:32.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:58:32.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:32.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:58:32.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:58:32.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:32.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:32.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:32.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:32.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:32.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:32.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:58:32.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:58:32.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:32.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:32.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:32.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:58:32.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:32.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:58:32.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:58:32.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:32.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:32.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:32.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:32.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:32.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:32.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:32.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:32.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:58:32.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:58:32.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:32.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:58:32.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:58:32.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:58:32.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:32.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:58:34.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:58:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:58:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:58:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:58:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:58:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:58:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:34.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:34.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:34.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:34.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:34.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:34.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:34.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:34.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:58:34.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:58:34.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:58:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:58:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:34.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:58:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:35.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:35.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:58:35.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:35.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:35.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:35.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:35.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:35.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:33 2026-02-22T19:58:35.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:35.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:58:35.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:35.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:35.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:32' 2026-02-22T19:58:35.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:58:35.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:58:35.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:35.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:58:35.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:35.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:35.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:35.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:35.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:35.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:35.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:35.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:35.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:33' 2026-02-22T19:58:35.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:58:33' 2026-02-22T19:58:35.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:35.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:35.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:35.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:35.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:33' 2026-02-22T19:58:35.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:35.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:35.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:58:33cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:58:32' 2026-02-22T19:58:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:35.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:35.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:33' 2026-02-22T19:58:35.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:35.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:58:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:35.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:58:35.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:58:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:35.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:35.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:36.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:58:36.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:36.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:36.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:36.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:36.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:36.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:36.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:36.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:36.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:58:36.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:32 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:36.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:33' 2026-02-22T19:58:36.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:58:36.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:58:36.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:36.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:58:36.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:36.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:36.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:36.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:36.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:36.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:32' 2026-02-22T19:58:36.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:58:32' 2026-02-22T19:58:36.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:36.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:36.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:36.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:36.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:32' 2026-02-22T19:58:36.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:36.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:36.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:58:32cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:58:33' 2026-02-22T19:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:36.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:32' 2026-02-22T19:58:36.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:36.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:58:36.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:36.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:58:36.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:36.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:36.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:36.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T19:58:36.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:36.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:36.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:36.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:58:36.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T19:58:37.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:58:37.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T19:58:37.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:37.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:37.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:37.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T19:58:37.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:37.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:37.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:37.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:58:37.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:58:37.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:37.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:38.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:58:38.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:38.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:38.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:38.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:58:38.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:38.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:58:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:58:38.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:58:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:58:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:38.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:58:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:58:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:58:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:58:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:58:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:58:38.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:58:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:58:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:58:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:58:38.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:58:38.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:58:38.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:38.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:58:38.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:58:38.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:38.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:38.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:58:38.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:38.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:58:38.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:58:38.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:58:38.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:38.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:38.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:58:38.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:38.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:38.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:38.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:38.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:58:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:58:38.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:38.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:58:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:58:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:58:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T19:58:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:58:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:58:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:58:38.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T19:58:38.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:38.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:38.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:38.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T19:58:41.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 59 2026-02-22T19:58:41.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T19:58:41.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:58:41.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:41.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:41.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:41.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:58:41.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:58:41.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:41.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:41.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:41.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:58:41.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:41.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:41.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:41.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:41.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:41.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:41.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:41.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:58:41.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:58:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:58:41.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=59 2026-02-22T19:58:41.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:58:41.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:41.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:58:41.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:41.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:41.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:41.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:41.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:41.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:58:41.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:58:41.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:58:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=59 2026-02-22T19:58:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 59 = 59 2026-02-22T19:58:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:58:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:41.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:41.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:41.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:42.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:58:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:38 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790316,"replay_state":"idle"} 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:37 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 58 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.bba77f3f-1d41-43fd-93ae-57fcec3da31a (peer_uuids:[]) 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 59 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.0c16cc81-809e-4d78-a3a0-319348d9cbef (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T19:58:42.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:58:42.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:58:42.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:42.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:58:42.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:42.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:58:42.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:42.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:58:42.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:42.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:42.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:42.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:42.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:38' 2026-02-22T19:58:42.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:58:38' 2026-02-22T19:58:42.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:42.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:42.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:42.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:42.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:38' 2026-02-22T19:58:42.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:42.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:42.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:42.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 19:58:38cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790316,"replay_state":"idle"}2026-02-22 19:58:37' 2026-02-22T19:58:42.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:42.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:42.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:38' 2026-02-22T19:58:42.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:42.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:58:42.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:58:42.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:58:42.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:58:42.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:42.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:42.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T19:58:42.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:42.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:42.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:42.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:42.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:42.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:42.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:42.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:43.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T19:58:43.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:43.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:43.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:43.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:43.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:43.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:43.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:43.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:43.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:58:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:58:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790316,"replay_state":"idle"} 2026-02-22T19:58:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:37 2026-02-22T19:58:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:58:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:58:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:58:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:43' 2026-02-22T19:58:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:58:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:58:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790316,"replay_state":"idle"} 2026-02-22T19:58:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:58:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:43.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:58:43.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:43.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:43.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:43.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:37' 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 19:58:37' 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:37' 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790316,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 19:58:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 19:58:43' 2026-02-22T19:58:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:37' 2026-02-22T19:58:44.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:44.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:58:44.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:58:44.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:58:44.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:58:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:44.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T19:58:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T19:58:45.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:58:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T19:58:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T19:58:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:58:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:58:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:45.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:46.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:58:46.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:46.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:46.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:46.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:58:46.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:46.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:58:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:58:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:46.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:58:46.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:58:46.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:46.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:46.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:46.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:46.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:58:46.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:46.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:46.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:58:46.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:58:46.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:46.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:46.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:58:46.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:58:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:46.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:58:46.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:46.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:46.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:58:46.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:58:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:58:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:58:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:58:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:58:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:47.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:47.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:47.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:47.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:47.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:58:47.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:47.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:47.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:58:47.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:58:47.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:47.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:43 2026-02-22T19:58:47.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:47.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:58:47.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:47.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:58:47.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:46' 2026-02-22T19:58:47.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:58:47.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T19:58:47.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:47.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:58:49.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:58:49.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:58:49.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:49.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:49.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:49.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:49.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:49.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:48 2026-02-22T19:58:49.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:49.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:58:49.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:49.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:49.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:47' 2026-02-22T19:58:49.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:58:49.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:58:49.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:49.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:58:49.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:49.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:49.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:49.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:49.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:49.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:49.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:49.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:49.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:48' 2026-02-22T19:58:49.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:58:48' 2026-02-22T19:58:49.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:49.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:49.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:49.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:49.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:48' 2026-02-22T19:58:49.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:49.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:49.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:49.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:58:48cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:58:47' 2026-02-22T19:58:49.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:49.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:49.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:48' 2026-02-22T19:58:49.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:49.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:58:49.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:49.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:58:49.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:49.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:49.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:49.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:58:49.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:49.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:49.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:49.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:49.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:49.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:49.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:49.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:47 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:48' 2026-02-22T19:58:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:58:50.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:58:50.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:58:50.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:58:50.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:58:50.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:50.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:58:50.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:50.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:58:50.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:58:50.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:58:50.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:58:50.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:47' 2026-02-22T19:58:50.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:58:47' 2026-02-22T19:58:50.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:50.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:50.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:50.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:58:50.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:47' 2026-02-22T19:58:50.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:58:50.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:58:50.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:58:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:58:47cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:58:48' 2026-02-22T19:58:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:58:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:58:50.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:47' 2026-02-22T19:58:50.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:58:50.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:58:50.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:58:50.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:58:50.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:58:50.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:50.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:50.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T19:58:50.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:50.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:50.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:50.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:58:50.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T19:58:52.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:58:52.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:58:52.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:52.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:52.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:58:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:58:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:58:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:58:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:58:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:58:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:58:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:58:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:58:53.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:58:53.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:53.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:53.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:53.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:53.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:58:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:58:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:58:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:53.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:53.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:58:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:58:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:58:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:53.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:53.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:53.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:53.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:53.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:53.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:58:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:58:53.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:53.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T19:58:53.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:53.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:58:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:58:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:58:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:58:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:58:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:58:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:58:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:58:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:58:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:58:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:58:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:58:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:58:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:58:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:58:55.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:55.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:55.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:58:55.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:58:55.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:58:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:58:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:58:55.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:58:55.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:58:55.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:58:55.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:58:55.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:55.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:58:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:58:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:58:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:58:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:58:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:58:59.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 62 2026-02-22T19:58:59.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:58:59.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:58:59.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:59.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:59.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:59.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:58:59.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:58:59.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:59.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:59.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:58:59.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:58:59.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:58:59.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:59.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:59.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:59.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:59.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:59.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:58:59.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:58:59.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:58:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:58:59.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=62 2026-02-22T19:58:59.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:58:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:58:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:58:59.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:59.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:58:59.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:59.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:59.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:58:59.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:58:59.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:58:59.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=62 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 62 = 62 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:58:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T19:59:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:00.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:00.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:00.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:59:00.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:00.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:00.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:59:00.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:59:00.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:00.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:57 2026-02-22T19:59:00.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:00.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:59:00.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:59:00.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790331,"replay_state":"idle"} 2026-02-22T19:59:00.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:53 2026-02-22T19:59:00.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:59:00.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 62 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.5622fb94-4bbe-4e6c-8a64-3d7ef4050c79 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:59:00.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:59:00.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:59:00.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:59:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:00.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:59:00.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:00.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:59:00.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:00.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:00.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:00.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:57' 2026-02-22T19:59:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 19:58:57' 2026-02-22T19:59:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:57' 2026-02-22T19:59:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:00.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:00.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 19:58:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790331,"replay_state":"idle"}2026-02-22 19:58:53' 2026-02-22T19:59:00.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:00.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:57' 2026-02-22T19:59:00.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:59:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:59:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:59:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:59:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:59:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:00.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:01.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:59:01.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:01.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:01.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:01.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:01.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:01.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:01.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:01.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:01.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:59:01.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:01.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:01.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:59:01.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790331,"replay_state":"idle"} 2026-02-22T19:59:01.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:01.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:53 2026-02-22T19:59:01.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:01.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:59:01.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:59:01.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:59:01.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:58:57' 2026-02-22T19:59:01.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:59:01.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:59:01.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:01.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790331,"replay_state":"idle"} 2026-02-22T19:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:59:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:01.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:01.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:01.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:01.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:53' 2026-02-22T19:59:01.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:58:53' 2026-02-22T19:59:01.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:01.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:01.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:01.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:01.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:58:53' 2026-02-22T19:59:01.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:01.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:01.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:01.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790331,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:58:53cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:58:57' 2026-02-22T19:59:01.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:01.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:01.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:58:53' 2026-02-22T19:59:01.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:01.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:59:01.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:59:01.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:59:01.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:59:01.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:01.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:01.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T19:59:01.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T19:59:01.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:01.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:01.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:01.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T19:59:02.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:59:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:59:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:59:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:59:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:59:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:59:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:03.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:59:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:59:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:59:03.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:03.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:03.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:59:03.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:59:03.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:03.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:59:03.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:59:03.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:59:03.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:03.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:59:05.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:59:05.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:05.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:05.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:05.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:59:05.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:05.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:05.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:59:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:59:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:59:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:59:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:05.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:05.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:05.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:05.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:05.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:05.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:59:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:59:05.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:05.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:59:05.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:05.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:59:05.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:05.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:05.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:05.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:05.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:05.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:05.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:05.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:06.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:59:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:06.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:06.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:59:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:59:06.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:03 2026-02-22T19:59:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:59:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:02' 2026-02-22T19:59:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:59:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:59:06.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:06.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:59:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:06.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:06.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:06.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:06.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:03' 2026-02-22T19:59:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:59:03' 2026-02-22T19:59:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:03' 2026-02-22T19:59:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:06.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:06.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 19:59:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:59:02' 2026-02-22T19:59:06.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:06.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:06.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:03' 2026-02-22T19:59:06.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:06.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:59:06.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:06.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:06.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:07.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:59:07.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:07.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:07.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:07.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:07.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:07.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:07.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:07.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:07.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:02 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:03' 2026-02-22T19:59:07.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:59:07.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:59:07.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:07.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:59:07.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:07.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:07.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:07.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:07.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:07.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:07.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:07.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:07.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:02' 2026-02-22T19:59:07.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:59:02' 2026-02-22T19:59:07.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:07.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:07.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:07.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:07.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:02' 2026-02-22T19:59:07.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:07.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:07.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:07.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:59:02cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 19:59:03' 2026-02-22T19:59:07.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:07.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:07.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:02' 2026-02-22T19:59:07.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:07.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:59:07.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:07.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:59:07.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:07.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:07.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:07.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T19:59:07.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:07.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:07.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:07.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:59:07.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T19:59:08.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:59:08.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T19:59:08.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:08.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:08.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:08.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T19:59:08.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:08.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:08.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:08.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:59:08.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:59:08.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:08.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:59:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:59:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:59:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:59:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:59:09.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:09.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:09.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:59:09.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:59:09.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:59:09.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:09.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:09.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:59:09.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:59:09.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:09.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T19:59:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:09.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:09.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:09.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T19:59:13.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 65 2026-02-22T19:59:13.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T19:59:13.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:59:13.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:13.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:13.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:13.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:59:13.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:59:13.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:13.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:13.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:13.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:59:13.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:59:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:59:13.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:59:13.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=65 2026-02-22T19:59:13.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:59:13.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:13.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:59:13.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:13.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:13.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:13.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:13.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:13.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:59:13.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:59:13.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=65 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 65 = 65 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:13.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:59:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:14.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:59:14.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:14.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:14.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:13 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790347,"replay_state":"idle"} 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:07 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 65 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.784a48ed-347f-4e9c-ade7-8a0df880f328 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:59:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:59:14.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:14.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:59:14.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:14.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:59:14.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:59:14.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:14.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:13' 2026-02-22T19:59:14.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:59:13' 2026-02-22T19:59:14.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:14.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:14.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:14.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:14.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:13' 2026-02-22T19:59:14.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:14.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:14.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 19:59:13cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790347,"replay_state":"idle"}2026-02-22 19:59:07' 2026-02-22T19:59:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:13' 2026-02-22T19:59:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:14.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:59:14.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:59:14.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:59:14.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:59:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T19:59:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T19:59:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:15.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:15.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:15.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:15.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:15.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:15.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:15.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:59:15.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:15.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:15.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:59:15.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790347,"replay_state":"idle"} 2026-02-22T19:59:15.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:15.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:07 2026-02-22T19:59:15.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:15.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:59:15.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:59:15.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:59:15.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:13' 2026-02-22T19:59:15.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:59:15.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:59:15.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:15.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790347,"replay_state":"idle"} 2026-02-22T19:59:15.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:15.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:59:15.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:15.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:59:15.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:15.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:15.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:15.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:15.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:07' 2026-02-22T19:59:15.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 19:59:07' 2026-02-22T19:59:15.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:07' 2026-02-22T19:59:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:15.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:15.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790347,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 19:59:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 19:59:13' 2026-02-22T19:59:15.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:15.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:15.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:07' 2026-02-22T19:59:15.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:15.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:59:15.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:59:15.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:59:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:59:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T19:59:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T19:59:16.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:59:16.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T19:59:16.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:16.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:16.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:16.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T19:59:16.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:16.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:16.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:16.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:59:16.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:59:16.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:16.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:17.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:59:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:59:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:59:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:59:17.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:59:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:59:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:17.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:17.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:59:17.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:17.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:17.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:59:17.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:59:17.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:17.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:17.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:59:17.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:59:17.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:17.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:17.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:59:17.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:17.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:17.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:59:17.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:17.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:17.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:59:17.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:59:17.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:17.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:17.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:59:17.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:17.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:17.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:17.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:17.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:59:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:59:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:59:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:59:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:59:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:59:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:59:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:59:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:19.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:59:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:59:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:59:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:59:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:19.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:59:19.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:19.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:19.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:59:19.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:59:19.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:19.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:19.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:59:19.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:19.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:59:19.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:19.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:19.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:19.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:19.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:59:19.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:59:19.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:19.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:59:19.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:19.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:59:19.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:19.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:19.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:19.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:19.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:19.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:19.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:19.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:20.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:59:20.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:20.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:20.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:20.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:20.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:20.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:20.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:20.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:20.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:59:20.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:20.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:20.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:18 2026-02-22T19:59:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:59:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:59:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:18' 2026-02-22T19:59:20.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:59:20.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:59:20.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:20.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:59:20.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:20.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:20.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:20.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:20.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:20.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:20.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:18' 2026-02-22T19:59:20.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:59:18' 2026-02-22T19:59:20.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:20.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:20.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:20.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:20.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:18' 2026-02-22T19:59:20.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:20.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:20.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:20.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:59:18cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 19:59:18' 2026-02-22T19:59:20.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:20.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:20.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:18' 2026-02-22T19:59:20.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:20.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:59:20.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:20.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:59:20.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:59:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:20.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:59:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:21.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:21.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:18 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:18' 2026-02-22T19:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:59:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:59:21.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T19:59:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:21.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:21.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:21.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:21.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:21.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:21.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:18' 2026-02-22T19:59:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:59:18' 2026-02-22T19:59:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:18' 2026-02-22T19:59:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:21.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:21.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 19:59:18cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:59:18' 2026-02-22T19:59:21.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:21.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:21.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:18' 2026-02-22T19:59:21.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:21.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:59:21.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:21.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:59:21.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:21.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:21.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:21.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T19:59:21.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:21.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:21.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:21.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:59:21.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T19:59:23.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:59:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:59:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:23.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:59:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:59:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:59:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:23.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:59:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:59:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:59:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:59:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:59:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:59:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:24.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:24.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:24.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:59:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:59:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:24.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:24.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:59:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:59:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:24.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:59:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:59:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:59:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:59:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:59:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:59:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:59:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:59:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:59:28.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 68 2026-02-22T19:59:28.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:59:28.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:59:28.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:28.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:28.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:28.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:28.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:59:28.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:28.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:59:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:59:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:28.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:28.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:28.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:28.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:28.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:28.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:28.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:59:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:59:28.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:59:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=68 2026-02-22T19:59:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:59:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:59:28.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:28.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:28.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:28.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:28.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:28.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:59:28.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:59:28.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:59:28.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=68 2026-02-22T19:59:28.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 68 = 68 2026-02-22T19:59:28.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:28.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T19:59:28.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:28.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:28.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:28.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:29.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T19:59:29.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:29.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:27 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790362,"replay_state":"idle"} 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:23 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 68 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.7c2df505-b280-4313-b732-5a215cddb3ca (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T19:59:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:59:29.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:59:29.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:59:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:59:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:59:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:29.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:29.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:29.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:27' 2026-02-22T19:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 19:59:27' 2026-02-22T19:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:27' 2026-02-22T19:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:29.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 19:59:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790362,"replay_state":"idle"}2026-02-22 19:59:23' 2026-02-22T19:59:29.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:29.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:29.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:27' 2026-02-22T19:59:29.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:29.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:59:29.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:59:29.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T19:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:29.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:29.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:29.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:29.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:30.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T19:59:30.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:30.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:30.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:30.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:30.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:30.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:30.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:30.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:30.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790362,"replay_state":"idle"} 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:23 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:27' 2026-02-22T19:59:30.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:59:30.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:59:30.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:30.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790362,"replay_state":"idle"} 2026-02-22T19:59:30.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:30.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:59:30.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:30.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:59:30.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:30.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:30.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:30.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:30.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:23' 2026-02-22T19:59:30.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:59:23' 2026-02-22T19:59:30.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:30.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:30.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:30.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:30.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:23' 2026-02-22T19:59:30.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:30.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:30.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790362,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:59:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:59:27' 2026-02-22T19:59:30.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:30.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:30.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:23' 2026-02-22T19:59:30.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:30.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:59:30.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:59:30.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:59:30.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:59:30.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:30.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:30.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T19:59:30.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T19:59:30.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:30.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:30.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:30.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T19:59:31.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:59:31.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T19:59:31.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T19:59:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:59:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:59:31.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:31.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:32.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:59:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:59:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:32.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:59:32.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:32.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:59:32.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:32.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:59:32.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:59:32.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:32.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:32.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:32.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:32.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:59:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:59:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:59:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:59:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:59:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:59:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:32.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:32.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:32.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:32.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:32.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:32.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:59:32.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:59:32.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:32.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:59:32.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:59:32.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:59:32.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:32.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:59:34.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T19:59:34.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:34.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:34.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:34.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:59:34.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:34.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:34.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:59:34.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:34.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:59:34.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:34.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:59:34.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:59:34.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:34.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:34.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:34.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:34.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:34.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:59:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:59:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:34.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:34.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:59:34.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:59:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:34.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:59:34.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:34.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:59:34.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:34.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:34.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:34.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:34.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:34.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:34.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:34.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:35.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:59:35.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:35.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:35.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:35.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:35.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:35.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:35.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:35.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:35.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:59:35.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:35.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:35.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:35.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:35.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:35.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:33 2026-02-22T19:59:35.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:35.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:59:35.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:35.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:35.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:32' 2026-02-22T19:59:35.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:59:35.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:59:35.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:35.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:59:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:35.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:35.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:35.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:35.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:33' 2026-02-22T19:59:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:59:33' 2026-02-22T19:59:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:33' 2026-02-22T19:59:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:59:33cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:59:32' 2026-02-22T19:59:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:35.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:35.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:33' 2026-02-22T19:59:35.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:35.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:59:35.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:35.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:59:35.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:59:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:35.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:36.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:59:36.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:36.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:36.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:36.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:36.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:36.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:36.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:36.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:36.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:59:36.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:36.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:36.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:36.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:36.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:36.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:32 2026-02-22T19:59:36.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:36.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:59:36.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:36.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:36.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:33' 2026-02-22T19:59:36.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:59:36.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:59:36.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:36.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:59:36.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:36.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:36.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:36.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:36.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:36.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:32' 2026-02-22T19:59:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:59:32' 2026-02-22T19:59:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:32' 2026-02-22T19:59:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:36.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:59:32cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:59:33' 2026-02-22T19:59:36.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:36.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:36.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:32' 2026-02-22T19:59:36.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:36.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:59:36.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:36.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:59:36.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:36.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T19:59:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:59:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T19:59:37.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:59:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T19:59:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T19:59:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:37.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:59:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:59:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:37.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:38.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:38.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:59:38.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:38.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:38.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:59:38.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:38.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:38.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:38.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:38.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:38.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:59:38.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:59:38.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:38.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:38.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:59:38.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:38.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:38.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:38.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:38.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:38.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:59:38.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:59:38.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:38.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:59:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:59:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:59:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T19:59:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:59:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:59:38.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:59:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T19:59:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T19:59:42.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 71 2026-02-22T19:59:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T19:59:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T19:59:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:42.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:59:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:59:42.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:42.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:42.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:42.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:59:42.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:42.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:42.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:42.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:42.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:42.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:42.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:42.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:59:42.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:59:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:59:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=71 2026-02-22T19:59:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:59:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:59:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:59:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:59:42.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:59:42.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=71 2026-02-22T19:59:42.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 71 = 71 2026-02-22T19:59:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T19:59:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:42.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T19:59:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:43.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:43 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790376,"replay_state":"idle"} 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:37 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 71 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.8b257d11-e4e9-4025-aea5-44d1b7c762c4 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T19:59:43.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T19:59:43.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T19:59:43.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:43.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T19:59:43.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:43.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:59:43.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:43.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:59:43.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:43.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:43.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:43.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:43' 2026-02-22T19:59:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 19:59:43' 2026-02-22T19:59:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:43' 2026-02-22T19:59:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:43.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:43.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:43.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 19:59:43cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790376,"replay_state":"idle"}2026-02-22 19:59:37' 2026-02-22T19:59:43.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:43.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:43.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:43' 2026-02-22T19:59:43.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:43.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T19:59:43.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T19:59:43.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T19:59:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T19:59:43.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:43.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:43.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T19:59:43.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:43.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:43.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:43.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:43.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:43.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:43.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:43.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:44.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T19:59:44.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:44.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:44.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:44.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:44.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:44.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790376,"replay_state":"idle"} 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:37 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:43' 2026-02-22T19:59:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T19:59:44.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T19:59:44.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:44.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790376,"replay_state":"idle"} 2026-02-22T19:59:44.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:44.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:59:44.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:44.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:59:44.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:44.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:44.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:44.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:44.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:37' 2026-02-22T19:59:44.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 19:59:37' 2026-02-22T19:59:44.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:44.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:44.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:44.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T19:59:44.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:37' 2026-02-22T19:59:44.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:44.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:44.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:45.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790376,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 19:59:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 19:59:43' 2026-02-22T19:59:45.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:45.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:45.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:37' 2026-02-22T19:59:45.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:45.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T19:59:45.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T19:59:45.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T19:59:45.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T19:59:45.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:45.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:45.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T19:59:45.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:45.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:45.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:45.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T19:59:45.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:45.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:59:46.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:59:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:59:46.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:46.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:46.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:46.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:46.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:46.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:59:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:59:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:46.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:59:46.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:59:46.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T19:59:46.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:46.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:59:48.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T19:59:48.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:48.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:48.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:48.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T19:59:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T19:59:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T19:59:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T19:59:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T19:59:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:48.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:48.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T19:59:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T19:59:48.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:48.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:48.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T19:59:48.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:48.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:48.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:59:48.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:59:48.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:48.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T19:59:48.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:48.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T19:59:48.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:48.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:48.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:48.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:48.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:48.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:48.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:48.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:49.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T19:59:49.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:49.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:49.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:49.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T19:59:49.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:49.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:49.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:49.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:49.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:49.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:48 2026-02-22T19:59:49.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:49.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T19:59:49.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:49.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:49.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:47' 2026-02-22T19:59:49.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:59:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:59:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:49.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:59:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:49.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:49.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:49.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:49.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:49.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:48' 2026-02-22T19:59:49.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 19:59:48' 2026-02-22T19:59:49.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:49.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:49.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:49.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:49.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:48' 2026-02-22T19:59:49.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:49.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:49.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:49.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 19:59:48cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 19:59:47' 2026-02-22T19:59:49.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:49.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:48' 2026-02-22T19:59:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:49.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:59:49.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:49.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:59:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:49.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:49.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:49.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T19:59:49.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:49.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:49.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:49.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:49.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:49.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:49.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:49.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T19:59:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:50.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:50.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:50.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T19:59:50.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T19:59:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T19:59:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T19:59:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:47 2026-02-22T19:59:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T19:59:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T19:59:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T19:59:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T19:59:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:48' 2026-02-22T19:59:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T19:59:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T19:59:50.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T19:59:50.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T19:59:50.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T19:59:50.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:50.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T19:59:50.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:50.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T19:59:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T19:59:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T19:59:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T19:59:50.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:47' 2026-02-22T19:59:50.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 19:59:47' 2026-02-22T19:59:50.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:50.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:50.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:50.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T19:59:50.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:47' 2026-02-22T19:59:50.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T19:59:50.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T19:59:50.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T19:59:50.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 19:59:47cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 19:59:48' 2026-02-22T19:59:50.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T19:59:50.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T19:59:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:47' 2026-02-22T19:59:50.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T19:59:50.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T19:59:50.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T19:59:50.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T19:59:50.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T19:59:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T19:59:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:50.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T19:59:50.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T19:59:52.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T19:59:52.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T19:59:52.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:52.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T19:59:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T19:59:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T19:59:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T19:59:53.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:59:53.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:59:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:59:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:59:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:59:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:59:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:53.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:59:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:59:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:59:53.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:59:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:53.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:53.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:53.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:53.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T19:59:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T19:59:53.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T19:59:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:53.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T19:59:55.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T19:59:55.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:55.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:55.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:55.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T19:59:55.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T19:59:55.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T19:59:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T19:59:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T19:59:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T19:59:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T19:59:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T19:59:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T19:59:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T19:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T19:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T19:59:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T19:59:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T19:59:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T19:59:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T19:59:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T19:59:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T19:59:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T19:59:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T19:59:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T19:59:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T19:59:55.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T19:59:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T19:59:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T19:59:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T19:59:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T19:59:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T19:59:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T19:59:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T19:59:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T19:59:55.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:55.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:55.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T19:59:55.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T19:59:55.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T19:59:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T19:59:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T19:59:55.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T19:59:55.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T19:59:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T19:59:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T19:59:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T19:59:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:59:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:55.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T19:59:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T19:59:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T19:59:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T19:59:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 74 2026-02-22T19:59:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T19:59:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T19:59:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T19:59:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T19:59:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T19:59:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T19:59:59.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T19:59:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T19:59:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T19:59:59.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:59:59.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T19:59:59.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=74 2026-02-22T19:59:59.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T19:59:59.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T19:59:59.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T19:59:59.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:59.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T19:59:59.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:59.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:59.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T19:59:59.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T19:59:59.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T19:59:59.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T19:59:59.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=74 2026-02-22T19:59:59.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 74 = 74 2026-02-22T19:59:59.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T19:59:59.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T19:59:59.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T19:59:59.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T19:59:59.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T19:59:59.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T19:59:59.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T19:59:59.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T19:59:59.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T19:59:59.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:00.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:00:00.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:00:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:00:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:00:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:57 2026-02-22T20:00:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:00:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:00:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790391,"replay_state":"idle"} 2026-02-22T20:00:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:53 2026-02-22T20:00:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:00:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 74 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.5e370371-f0bd-44c0-a5f2-24373f5d574c (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:00:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:00:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:00:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:00:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:00:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:00.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:00:00.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:00.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:00.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:00.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:00.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:57' 2026-02-22T20:00:00.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 19:59:57' 2026-02-22T20:00:00.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:00.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:57' 2026-02-22T20:00:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:00.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:00.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 19:59:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790391,"replay_state":"idle"}2026-02-22 19:59:53' 2026-02-22T20:00:00.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:00.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:00.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:57' 2026-02-22T20:00:00.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:00.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:00:00.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:00:00.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:00:00.468 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:00:00.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:00.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:00:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:01.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:00:01.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:01.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:01.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:01.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:01.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:01.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:01.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:01.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:01.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:00:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:00:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790391,"replay_state":"idle"} 2026-02-22T20:00:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:53 2026-02-22T20:00:01.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:01.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:00:01.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:00:01.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:00:01.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 19:59:57' 2026-02-22T20:00:01.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:00:01.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:00:01.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:01.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790391,"replay_state":"idle"} 2026-02-22T20:00:01.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:01.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:00:01.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:01.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:00:01.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:01.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:53' 2026-02-22T20:00:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 19:59:53' 2026-02-22T20:00:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 19:59:53' 2026-02-22T20:00:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:01.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790391,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 19:59:53cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 19:59:57' 2026-02-22T20:00:01.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:01.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:01.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 19:59:53' 2026-02-22T20:00:01.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:01.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:00:01.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:00:01.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:00:01.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:00:01.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:01.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:01.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:00:01.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:00:01.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:01.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:01.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:01.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:00:02.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:00:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:00:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:00:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:00:03.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:00:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:00:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:00:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:00:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:03.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:03.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:03.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:00:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:00:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:00:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:00:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:00:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:00:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:00:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:00:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:00:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:00:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:03.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:03.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:00:03.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:00:03.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:03.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:00:03.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:00:03.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:00:03.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:03.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:00:05.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:00:05.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:05.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:05.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:05.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:00:05.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:05.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:00:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:00:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:00:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:00:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:05.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:05.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:05.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:05.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:05.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:00:05.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:00:05.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:05.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:05.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:05.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:00:05.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:05.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:00:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:00:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:00:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:05.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:05.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:00:05.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:00:05.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:05.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:00:05.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:05.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:00:05.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:05.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:05.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:05.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:05.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:05.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:05.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:05.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:06.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:00:06.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:06.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:00:06.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:06.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:06.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:06.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:00:06.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:06.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:03 2026-02-22T20:00:06.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:06.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:00:06.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:06.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:06.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:02' 2026-02-22T20:00:06.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:00:06.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:00:06.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:06.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:00:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:06.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:06.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:06.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:06.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:06.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:06.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:03' 2026-02-22T20:00:06.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:00:03' 2026-02-22T20:00:06.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:06.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:06.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:06.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:06.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:03' 2026-02-22T20:00:06.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:06.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:06.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:06.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 20:00:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:00:02' 2026-02-22T20:00:06.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:06.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:06.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:03' 2026-02-22T20:00:06.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:00:06.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:06.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:00:06.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:06.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:06.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:06.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:00:06.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:06.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:06.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:06.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:06.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:06.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:06.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:06.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:00:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:07.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:00:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:02 2026-02-22T20:00:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:00:07.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:07.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:00:07.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:03' 2026-02-22T20:00:07.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:00:07.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:00:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:00:07.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:07.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:07.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:07.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:07.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:07.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:07.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:07.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:02' 2026-02-22T20:00:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:00:02' 2026-02-22T20:00:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:02' 2026-02-22T20:00:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:07.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:07.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:00:02cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 20:00:03' 2026-02-22T20:00:07.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:07.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:07.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:02' 2026-02-22T20:00:07.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:07.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:00:07.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:07.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:00:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:00:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:00:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:00:08.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:00:08.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:00:08.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:08.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:08.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:08.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:00:08.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:08.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:08.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:08.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:00:08.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:00:08.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:08.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:00:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:09.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:09.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:09.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:00:09.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:09.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:09.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:00:09.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:09.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:00:09.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:09.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:00:09.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:00:09.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:09.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:09.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:09.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:00:09.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:09.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:09.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:00:09.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:00:09.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:09.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:09.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:00:09.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:00:09.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:09.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:09.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:00:09.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:00:09.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:09.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:09.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:09.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:09.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:09.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:00:09.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:00:09.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:09.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:09.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:00:09.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:09.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:09.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:09.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:09.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:09.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:00:09.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:00:09.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:09.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:00:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:00:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:00:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:00:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:00:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:00:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:00:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:00:09.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:09.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:09.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:09.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:00:13.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 77 2026-02-22T20:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:13.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:00:13.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:00:13.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:13.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:13.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:13.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:00:13.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:13.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:13.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:13.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:13.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:13.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:13.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:13.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:00:13.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:00:13.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:00:13.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=77 2026-02-22T20:00:13.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:00:13.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:13.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:00:13.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:13.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:13.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:13.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:13.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:13.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:00:13.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:00:13.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:00:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=77 2026-02-22T20:00:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 77 = 77 2026-02-22T20:00:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:00:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:13.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:13.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:13.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:13.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:13.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:14.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:00:14.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:14.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:14.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:14.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:14.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:14.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:14.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:14.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:14.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:00:14.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:00:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:00:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:13 2026-02-22T20:00:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:00:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:00:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790407,"replay_state":"idle"} 2026-02-22T20:00:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:07 2026-02-22T20:00:14.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:00:14.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 77 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.06719d54-5060-4d90-af71-7eb39f395056 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:00:14.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:00:14.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:00:14.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:00:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:00:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:00:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:14.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:14.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:14.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:13' 2026-02-22T20:00:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:00:13' 2026-02-22T20:00:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:13' 2026-02-22T20:00:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:15.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:00:13cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790407,"replay_state":"idle"}2026-02-22 20:00:07' 2026-02-22T20:00:15.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:15.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:15.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:13' 2026-02-22T20:00:15.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:15.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:00:15.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:00:15.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:00:15.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:00:15.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:15.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:15.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:00:15.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:15.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:15.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:15.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:15.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:15.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:15.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:15.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:16.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:00:16.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:16.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:16.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:00:16.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790407,"replay_state":"idle"} 2026-02-22T20:00:16.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:16.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:07 2026-02-22T20:00:16.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:00:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:00:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:00:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:13' 2026-02-22T20:00:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:00:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:00:16.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:16.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790407,"replay_state":"idle"} 2026-02-22T20:00:16.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:16.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:00:16.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:16.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:00:16.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:16.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:16.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:16.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:16.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:07' 2026-02-22T20:00:16.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:00:07' 2026-02-22T20:00:16.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:16.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:16.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:07' 2026-02-22T20:00:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:16.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790407,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:00:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:00:13' 2026-02-22T20:00:16.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:16.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:16.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:07' 2026-02-22T20:00:16.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:16.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:00:16.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:00:16.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:00:16.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:00:16.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:16.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:16.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:00:16.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:16.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:16.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:16.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:00:16.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:00:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:00:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:00:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:16.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:00:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:00:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:16.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:00:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:00:17.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:00:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:00:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:00:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:00:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:00:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:00:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:00:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:00:17.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:00:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:00:17.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:17.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:17.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:17.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:17.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:00:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:00:17.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:00:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:00:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:00:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:00:19.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:00:19.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:19.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:19.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:19.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:00:19.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:19.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:19.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:19.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:00:19.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:00:19.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:19.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:00:19.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:00:19.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:19.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:19.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:19.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:00:19.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:19.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:19.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:00:19.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:00:19.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:19.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:19.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:00:19.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:19.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:19.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:00:19.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:19.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:00:19.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:00:19.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:19.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:00:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:00:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:20.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:00:20.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:20.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:20.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:20.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:20.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:20.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:20.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:20.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:20.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:00:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:18 2026-02-22T20:00:20.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:00:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:00:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:18' 2026-02-22T20:00:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:00:20.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:00:20.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:20.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:00:20.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:20.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:20.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:20.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:20.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:20.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:20.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:20.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:18' 2026-02-22T20:00:20.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:00:18' 2026-02-22T20:00:20.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:20.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:20.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:20.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:20.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:18' 2026-02-22T20:00:20.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:20.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:20.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:20.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:00:18cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 20:00:18' 2026-02-22T20:00:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:20.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:20.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:18' 2026-02-22T20:00:20.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:20.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:00:20.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:20.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:00:20.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:00:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:20.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:20.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:20.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:20.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:20.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:21.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:00:21.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:21.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:21.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:21.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:21.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:21.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:21.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:21.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:21.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:00:21.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:21.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:21.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:21.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:00:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:18 2026-02-22T20:00:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:00:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:18' 2026-02-22T20:00:21.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:00:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:00:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:21.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:00:21.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:21.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:21.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:21.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:21.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:21.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:21.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:21.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:18' 2026-02-22T20:00:21.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:00:18' 2026-02-22T20:00:21.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:21.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:21.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:21.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:21.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:18' 2026-02-22T20:00:21.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:21.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:21.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:21.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 20:00:18cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:00:18' 2026-02-22T20:00:21.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:21.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:21.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:18' 2026-02-22T20:00:21.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:21.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:00:21.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:21.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:00:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:00:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:00:21.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:00:23.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:00:23.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:00:23.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:23.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:23.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:23.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:00:23.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:23.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:23.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:00:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:00:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:00:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:24.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:24.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:24.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:24.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:24.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:00:24.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:00:24.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:24.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:24.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:24.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:00:24.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:24.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:24.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:00:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:00:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:00:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:24.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:24.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:24.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:24.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:24.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:00:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:00:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:24.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:00:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:00:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:00:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:00:24.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:00:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:00:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:00:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:00:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:24.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:00:28.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 80 2026-02-22T20:00:28.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:00:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:00:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:28.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:00:28.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:28.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:28.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:00:28.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:00:28.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:28.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:28.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:28.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:28.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:28.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:28.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:28.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:00:28.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:00:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:00:28.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=80 2026-02-22T20:00:28.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:00:28.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:28.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:00:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:28.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:28.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:00:28.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:00:28.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:00:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=80 2026-02-22T20:00:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 80 = 80 2026-02-22T20:00:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:00:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:28.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:29.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:00:29.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:29.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:29.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:29.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:29.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:29.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:29.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:29.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:29.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:27 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790422,"replay_state":"idle"} 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:23 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 80 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.8171b555-4061-4efc-887f-c4010c9ceaeb (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:00:29.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:00:29.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:00:29.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:29.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:00:29.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:29.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:00:29.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:29.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:00:29.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:29.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:27' 2026-02-22T20:00:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:00:27' 2026-02-22T20:00:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:27' 2026-02-22T20:00:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:00:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790422,"replay_state":"idle"}2026-02-22 20:00:23' 2026-02-22T20:00:29.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:29.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:27' 2026-02-22T20:00:29.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:00:29.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:00:29.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:00:29.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:00:29.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:29.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:29.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:00:29.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:29.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:29.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:29.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:30.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:00:30.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:30.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:00:30.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:30.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:30.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:00:30.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790422,"replay_state":"idle"} 2026-02-22T20:00:30.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:30.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:23 2026-02-22T20:00:30.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:30.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:00:30.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:00:30.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:00:30.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:27' 2026-02-22T20:00:30.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:00:30.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:00:30.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790422,"replay_state":"idle"} 2026-02-22T20:00:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:00:30.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:30.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:00:30.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:30.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:30.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:30.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:30.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:23' 2026-02-22T20:00:30.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:00:23' 2026-02-22T20:00:30.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:30.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:30.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:30.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:30.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:23' 2026-02-22T20:00:30.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:30.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:30.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:30.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790422,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:00:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:00:27' 2026-02-22T20:00:30.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:30.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:30.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:23' 2026-02-22T20:00:30.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:30.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:00:30.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:00:30.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:00:30.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:00:30.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:30.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:30.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:00:30.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:00:30.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:30.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:30.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:30.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:00:31.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:00:31.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:00:31.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:31.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:31.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:31.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:00:31.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:31.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:31.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:31.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:00:31.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:00:31.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:31.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:00:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:32.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:32.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:32.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:32.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:00:32.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:32.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:32.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:32.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:00:32.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:00:32.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:32.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:32.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:00:32.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:32.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:32.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:32.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:32.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:32.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:00:32.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:00:32.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:32.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:32.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:32.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:32.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:32.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:32.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:32.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:32.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:00:32.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:00:32.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:32.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:00:32.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:00:32.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:00:32.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:32.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:00:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:00:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:34.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:00:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:34.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:00:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:00:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:00:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:00:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:34.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:34.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:34.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:34.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:34.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:34.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:00:34.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:00:34.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:34.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:34.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:34.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:00:34.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:00:34.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:34.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:34.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:34.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:34.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:34.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:34.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:00:34.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:00:34.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:34.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:00:34.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:34.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:00:34.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:34.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:34.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:34.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:34.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:34.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:34.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:34.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:00:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:35.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:35.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:00:35.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:33 2026-02-22T20:00:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:35.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:00:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:32' 2026-02-22T20:00:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:00:35.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:00:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:35.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:00:35.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:35.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:35.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:35.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:35.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:35.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:35.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:35.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:35.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:33' 2026-02-22T20:00:35.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:00:33' 2026-02-22T20:00:35.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:35.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:35.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:35.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:35.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:33' 2026-02-22T20:00:35.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:35.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:35.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:35.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:00:33cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:00:32' 2026-02-22T20:00:35.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:35.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:35.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:33' 2026-02-22T20:00:35.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:35.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:00:35.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:35.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:00:35.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:00:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:35.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:00:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:36.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:36.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:36.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:36.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:36.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:36.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:00:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:32 2026-02-22T20:00:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:00:36.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:33' 2026-02-22T20:00:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:00:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:00:36.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:36.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:00:36.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:36.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:36.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:36.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:36.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:36.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:36.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:32' 2026-02-22T20:00:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:00:32' 2026-02-22T20:00:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:32' 2026-02-22T20:00:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:36.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:00:32cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:00:33' 2026-02-22T20:00:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:36.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:36.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:32' 2026-02-22T20:00:36.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:00:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:00:36.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:36.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:36.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:36.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:00:36.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:36.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:36.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:36.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:00:36.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:00:37.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:00:37.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:00:37.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:37.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:37.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:37.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:00:37.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:37.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:37.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:37.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:00:37.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:00:37.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:37.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:38.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:00:38.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:38.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:38.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:38.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:00:38.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:38.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:38.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:00:38.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:00:38.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:38.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:38.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:00:38.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:00:38.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:38.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:38.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:38.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:00:38.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:00:38.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:00:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:00:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:00:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:38.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:38.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:38.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:38.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:38.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:00:38.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:00:38.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:38.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:00:38.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:00:38.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:00:38.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:38.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:00:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:00:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:00:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:00:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:00:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:00:42.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 83 2026-02-22T20:00:42.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:00:42.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:00:42.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:42.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:42.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:42.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:00:42.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:00:42.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:00:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:42.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:42.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:42.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:42.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:42.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:42.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:42.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:00:42.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:00:42.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:00:42.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=83 2026-02-22T20:00:42.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:00:42.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:42.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:00:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:42.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:00:42.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:00:42.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:00:42.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=83 2026-02-22T20:00:42.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 83 = 83 2026-02-22T20:00:42.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:42.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:00:42.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:42.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:42.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:42.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:42.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:42.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:42.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:42.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:43.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:00:43.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:43.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:43.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:43.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:43.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:43.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:43.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:43.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:43.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:00:43.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:00:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:00:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:43 2026-02-22T20:00:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:00:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:00:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790436,"replay_state":"idle"} 2026-02-22T20:00:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:37 2026-02-22T20:00:43.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:00:43.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 83 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.9e713396-9ee6-411f-9b17-f93b42b122ff (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:00:43.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:00:43.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:00:43.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:43.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:00:43.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:43.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:00:43.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:43.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:00:43.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:43' 2026-02-22T20:00:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:00:43' 2026-02-22T20:00:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:43' 2026-02-22T20:00:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:43.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:44.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:00:43cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790436,"replay_state":"idle"}2026-02-22 20:00:37' 2026-02-22T20:00:44.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:44.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:44.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:43' 2026-02-22T20:00:44.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:44.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:00:44.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:00:44.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:00:44.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:00:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:00:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:00:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:45.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:45.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:45.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:45.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:45.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:00:45.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:45.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:45.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:00:45.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790436,"replay_state":"idle"} 2026-02-22T20:00:45.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:45.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:37 2026-02-22T20:00:45.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:45.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:00:45.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:00:45.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:00:45.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:43' 2026-02-22T20:00:45.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:00:45.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:00:45.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:45.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790436,"replay_state":"idle"} 2026-02-22T20:00:45.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:45.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:00:45.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:45.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:00:45.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:45.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:45.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:45.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:45.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:37' 2026-02-22T20:00:45.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:00:37' 2026-02-22T20:00:45.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:45.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:45.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:45.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:00:45.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:37' 2026-02-22T20:00:45.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:45.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:45.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:45.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790436,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:00:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:00:43' 2026-02-22T20:00:45.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:45.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:45.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:37' 2026-02-22T20:00:45.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:00:45.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:00:45.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:00:45.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:00:45.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:45.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:45.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:00:45.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:45.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:45.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:45.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:00:45.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:00:45.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:00:45.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:45.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:45.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:45.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:00:45.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:45.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:45.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:45.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:00:45.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:00:45.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:45.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:46.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:00:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:00:46.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:00:46.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:00:46.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:46.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:46.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:00:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:46.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:46.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:00:46.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:00:46.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:46.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:00:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:00:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:00:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:46.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:00:48.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:00:48.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:48.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:48.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:48.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:00:48.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:48.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:48.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:00:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:00:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:00:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:00:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:48.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:48.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:00:48.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:00:48.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:00:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:00:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:00:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:00:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:00:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:48.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:00:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:00:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:48.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:48.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:00:48.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:00:48.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:00:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:00:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:48.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:49.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:00:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:49.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:48 2026-02-22T20:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:47' 2026-02-22T20:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:00:49.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:00:49.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:49.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:00:49.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:49.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:49.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:49.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:49.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:49.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:49.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:48' 2026-02-22T20:00:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:00:48' 2026-02-22T20:00:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:48' 2026-02-22T20:00:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:00:48cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:00:47' 2026-02-22T20:00:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:48' 2026-02-22T20:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:49.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:00:49.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:49.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:00:49.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:00:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:50.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:00:50.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:50.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:50.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:50.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:50.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:50.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:50.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:50.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:50.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:00:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:00:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:00:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:00:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:47 2026-02-22T20:00:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:00:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:00:50.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:00:50.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:00:50.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:48' 2026-02-22T20:00:50.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:00:50.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:00:50.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:00:50.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:00:50.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:00:50.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:50.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:00:50.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:50.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:00:50.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:00:50.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:00:50.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:00:50.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:47' 2026-02-22T20:00:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:00:47' 2026-02-22T20:00:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:00:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:47' 2026-02-22T20:00:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:00:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:00:50.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:00:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:00:47cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:00:48' 2026-02-22T20:00:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:00:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:00:51.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:47' 2026-02-22T20:00:51.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:00:51.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:00:51.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:00:51.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:00:51.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:00:51.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:51.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:51.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:00:51.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:51.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:51.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:51.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:00:51.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:00:52.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:00:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:00:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:00:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:00:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:00:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:00:53.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:00:53.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:53.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:53.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:53.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:53.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:53.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:53.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:53.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:53.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:53.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:53.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:00:53.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:00:53.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:53.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T20:00:53.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:53.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:00:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:00:55.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:55.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:55.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:55.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:00:55.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:00:55.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:00:55.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:00:55.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:00:55.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:00:55.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:55.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:00:55.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:00:55.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:00:55.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:55.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:00:55.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:00:55.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:00:55.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:00:55.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:00:55.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:00:55.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:00:55.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:00:55.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:00:55.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:00:55.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:00:55.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:00:55.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:00:55.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:00:55.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:00:55.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:00:55.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:00:55.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:00:55.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:55.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:00:55.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:00:55.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:55.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:55.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:55.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:00:55.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:00:55.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:00:55.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:55.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:55.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:55.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:00:59.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 86 2026-02-22T20:00:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:00:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:00:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:59.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:00:59.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:00:59.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:59.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:59.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:00:59.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:00:59.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:00:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:00:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:00:59.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:00:59.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:00:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=86 2026-02-22T20:00:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:00:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:00:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:00:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:00:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:00:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:00:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:00:59.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:00:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=86 2026-02-22T20:00:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 86 = 86 2026-02-22T20:00:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:00:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:00:59.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:00:59.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:00:59.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:00:59.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:00:59.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:00:59.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:00:59.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:00:59.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:01:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:01:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:01:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:01:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:57 2026-02-22T20:01:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:01:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:01:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790451,"replay_state":"idle"} 2026-02-22T20:01:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:53 2026-02-22T20:01:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:01:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 86 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.4a599d9d-63c0-4b3a-b825-caace821df2a (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:01:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:01:00.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:01:00.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:00.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:01:00.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:00.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:01:00.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:00.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:01:00.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:00.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:00.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:00.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:57' 2026-02-22T20:01:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:00:57' 2026-02-22T20:01:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:00.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:00.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:00.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:57' 2026-02-22T20:01:00.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:00.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:00.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:00.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:00:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790451,"replay_state":"idle"}2026-02-22 20:00:53' 2026-02-22T20:01:00.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:00.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:00.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:57' 2026-02-22T20:01:00.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:00.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:01:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:01:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:01:00.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:01:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:01:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:00.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:00.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:00.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:00.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:00.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:00.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:01.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:01:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:01.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790451,"replay_state":"idle"} 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:53 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:00:57' 2026-02-22T20:01:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:01:01.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:01:01.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:01.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790451,"replay_state":"idle"} 2026-02-22T20:01:01.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:01.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:01:01.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:01:01.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:01.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:53' 2026-02-22T20:01:01.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:00:53' 2026-02-22T20:01:01.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:00:53' 2026-02-22T20:01:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:01.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790451,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:00:53cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:00:57' 2026-02-22T20:01:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:01.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:00:53' 2026-02-22T20:01:01.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:01.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:01:01.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:01:01.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:01:01.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:01:01.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:01.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:01.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:01:01.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:01:01.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:01.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:01.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:01.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:01:02.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:03.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:01:03.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:03.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:03.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:03.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:01:03.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:03.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:01:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:01:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:01:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:01:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:03.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:03.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:01:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:01:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:01:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:03.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:03.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:03.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:03.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:01:03.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:01:03.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:03.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:01:03.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:01:03.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:01:03.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:03.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:01:05.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:01:05.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:05.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:05.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:05.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:01:05.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:05.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:05.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:01:05.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:05.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:01:05.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:05.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:01:05.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:01:05.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:05.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:05.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:05.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:05.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:05.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:01:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:01:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:05.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:05.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:05.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:05.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:05.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:01:05.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:01:05.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:05.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:01:05.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:05.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:01:05.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:05.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:05.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:05.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:05.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:05.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:05.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:05.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:06.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:01:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:06.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:01:06.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:03 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:02' 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:01:06.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:01:06.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:06.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:01:06.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:06.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:06.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:06.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:06.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:06.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:06.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:06.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:03' 2026-02-22T20:01:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:01:03' 2026-02-22T20:01:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:03' 2026-02-22T20:01:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:06.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 20:01:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:01:02' 2026-02-22T20:01:06.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:06.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:06.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:03' 2026-02-22T20:01:06.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:06.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:01:06.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:06.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:01:06.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:01:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:07.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:01:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:07.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:01:07.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:07.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:07.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:07.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:01:07.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:07.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:02 2026-02-22T20:01:07.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:07.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:01:07.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:07.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:01:07.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:03' 2026-02-22T20:01:07.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:01:07.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:01:07.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:01:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:07.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:07.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:07.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:07.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:07.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:07.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:02' 2026-02-22T20:01:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:01:02' 2026-02-22T20:01:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:07.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:02' 2026-02-22T20:01:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:07.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:01:02cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 20:01:03' 2026-02-22T20:01:07.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:07.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:07.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:02' 2026-02-22T20:01:07.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:07.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:01:07.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:07.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:01:07.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:01:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:07.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:07.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:01:07.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:01:08.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:01:08.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:01:08.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:08.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:08.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:08.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:01:08.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:08.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:08.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:08.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:01:08.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:01:08.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:08.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:09.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:01:09.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:09.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:09.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:09.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:01:09.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:09.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:09.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:01:09.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:09.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:01:09.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:09.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:01:09.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:01:09.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:09.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:09.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:09.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:01:09.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:09.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:09.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:01:09.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:09.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:09.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:09.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:09.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:09.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:01:09.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:01:09.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:09.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:01:09.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:01:09.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:01:09.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:09.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:01:09.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:01:09.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:09.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:09.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:09.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:01:09.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:01:09.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:01:09.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:09.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:09.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:09.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:01:13.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 89 2026-02-22T20:01:13.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:01:13.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:01:13.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:13.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:13.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:13.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:01:13.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:01:13.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:13.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:13.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:13.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:01:13.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:13.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:13.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:13.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:13.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:13.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:01:13.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:01:13.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:01:13.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=89 2026-02-22T20:01:13.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:01:13.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:13.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:01:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:13.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:13.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:01:13.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:01:13.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:01:13.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=89 2026-02-22T20:01:13.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 89 = 89 2026-02-22T20:01:13.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:01:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:14.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:01:14.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:13 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790467,"replay_state":"idle"} 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:07 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 89 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.b57bb9c4-c5c4-4fbb-bfb9-d4b71b6b7e9e (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:01:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:01:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:15.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:01:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:01:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:01:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:15.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:13' 2026-02-22T20:01:15.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:01:13' 2026-02-22T20:01:15.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:15.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:15.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:15.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:15.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:13' 2026-02-22T20:01:15.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:15.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:15.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:15.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:01:13cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790467,"replay_state":"idle"}2026-02-22 20:01:07' 2026-02-22T20:01:15.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:15.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:15.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:13' 2026-02-22T20:01:15.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:15.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:01:15.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:01:15.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:01:15.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:01:15.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:15.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:15.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:01:15.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:15.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:15.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:15.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:15.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:15.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:15.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:15.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:16.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:01:16.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:16.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:16.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:16.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:16.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:16.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:16.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:16.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:16.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790467,"replay_state":"idle"} 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:07 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:13' 2026-02-22T20:01:16.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:01:16.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:01:16.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:16.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790467,"replay_state":"idle"} 2026-02-22T20:01:16.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:16.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:01:16.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:16.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:01:16.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:16.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:16.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:16.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:16.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:07' 2026-02-22T20:01:16.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:01:07' 2026-02-22T20:01:16.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:16.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:16.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:16.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:16.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:07' 2026-02-22T20:01:16.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:16.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:16.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:16.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790467,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:01:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:01:13' 2026-02-22T20:01:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:16.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:07' 2026-02-22T20:01:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:16.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:01:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:01:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:01:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:01:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:01:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:01:16.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:01:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:01:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:01:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:01:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:01:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:01:17.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:17.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:17.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:17.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:01:17.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:17.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:17.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:01:17.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:17.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:01:17.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:17.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:01:17.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:01:17.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:17.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:17.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:17.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:01:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:01:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:01:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:01:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:01:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:01:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:01:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:01:17.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:01:17.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:17.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:17.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:01:17.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:17.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:17.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:17.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:17.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:17.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:01:17.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:01:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:17.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:01:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:01:17.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:01:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:17.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:01:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:01:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:01:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:01:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:01:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:01:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:01:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:19.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:01:19.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:19.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:19.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:19.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:19.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:01:19.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:19.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:19.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:19.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:19.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:19.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:01:19.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:01:19.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:01:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:01:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:19.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:19.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:19.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:19.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:19.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:01:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:20.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:20.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:20.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:18 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:18' 2026-02-22T20:01:20.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:01:20.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:01:20.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:20.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:01:20.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:20.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:20.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:20.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:20.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:20.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:18' 2026-02-22T20:01:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:01:18' 2026-02-22T20:01:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:18' 2026-02-22T20:01:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:20.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:20.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:20.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:01:18cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 20:01:18' 2026-02-22T20:01:20.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:20.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:20.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:18' 2026-02-22T20:01:20.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:20.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:01:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:01:20.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:01:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:21.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:01:21.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:21.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:21.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:21.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:21.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:21.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:21.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:21.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:21.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:18 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:18' 2026-02-22T20:01:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:01:22.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:01:22.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:22.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:01:22.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:22.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:22.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:22.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:22.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:22.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:22.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:22.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:18' 2026-02-22T20:01:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:01:18' 2026-02-22T20:01:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:18' 2026-02-22T20:01:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:22.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 20:01:18cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:01:18' 2026-02-22T20:01:22.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:22.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:22.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:18' 2026-02-22T20:01:22.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:22.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:01:22.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:22.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:01:22.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:22.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:22.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:22.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:01:22.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:22.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:22.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:22.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:01:22.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:01:24.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:01:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:01:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:01:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:24.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:01:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:01:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:25.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:01:25.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:25.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:25.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:25.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:01:25.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:25.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:25.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:01:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:01:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:01:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:01:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:25.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:25.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:01:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:25.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:25.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:25.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:25.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:25.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:25.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:25.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:25.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:01:25.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:01:25.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:25.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:01:25.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:01:25.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:01:25.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:25.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:01:25.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:01:25.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:25.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:25.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:25.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:01:25.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:01:25.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:01:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:01:29.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 92 2026-02-22T20:01:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:01:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:01:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:01:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:01:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:01:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:01:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:01:29.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:01:29.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=92 2026-02-22T20:01:29.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:01:29.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:29.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:01:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:29.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:01:29.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:01:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:01:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=92 2026-02-22T20:01:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 92 = 92 2026-02-22T20:01:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:01:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:01:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:01:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:01:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:01:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:27 2026-02-22T20:01:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:01:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:01:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790483,"replay_state":"idle"} 2026-02-22T20:01:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:23 2026-02-22T20:01:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:01:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 92 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.5295cd17-618f-4f27-b1b0-f26b4efb8ad4 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:01:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:01:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:01:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:30.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:01:30.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:30.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:01:30.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:30.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:01:30.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:30.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:30.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:30.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:30.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:27' 2026-02-22T20:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:01:27' 2026-02-22T20:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:27' 2026-02-22T20:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:30.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:01:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790483,"replay_state":"idle"}2026-02-22 20:01:23' 2026-02-22T20:01:30.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:30.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:30.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:27' 2026-02-22T20:01:30.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:30.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:01:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:01:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:01:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:01:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:01:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:30.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:31.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:01:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:31.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:01:31.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:31.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:31.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:01:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790483,"replay_state":"idle"} 2026-02-22T20:01:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:23 2026-02-22T20:01:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:01:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:01:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:01:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:27' 2026-02-22T20:01:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:01:31.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:01:31.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790483,"replay_state":"idle"} 2026-02-22T20:01:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:01:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:31.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:01:31.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:31.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:31.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:23' 2026-02-22T20:01:31.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:01:23' 2026-02-22T20:01:31.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:31.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:31.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:31.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:31.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:23' 2026-02-22T20:01:31.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:31.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:31.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:31.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790483,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:01:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:01:27' 2026-02-22T20:01:31.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:31.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:31.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:23' 2026-02-22T20:01:31.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:31.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:01:31.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:01:31.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:01:31.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:01:31.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:31.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:31.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:01:31.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:01:31.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:31.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:31.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:31.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:01:32.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:01:32.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:33.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:01:33.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:33.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:33.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:33.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:33.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:33.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:33.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:33.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:33.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:33.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:33.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:01:33.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:01:33.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:33.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:01:33.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:01:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:01:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:35.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:35.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:35.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:01:35.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:35.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:01:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:01:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:01:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:01:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:01:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:01:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:01:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:01:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:01:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:01:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:35.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:35.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:35.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:35.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:35.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:35.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:01:35.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:01:35.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:01:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:01:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:35.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:01:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:36.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:33 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:32' 2026-02-22T20:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:01:36.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:01:36.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:36.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:01:36.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:36.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:36.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:36.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:36.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:36.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:36.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:33' 2026-02-22T20:01:36.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:01:33' 2026-02-22T20:01:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:33' 2026-02-22T20:01:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:36.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:36.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 20:01:33cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:01:32' 2026-02-22T20:01:36.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:36.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:36.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:33' 2026-02-22T20:01:36.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:36.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:01:36.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:36.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:01:36.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:36.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:36.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:01:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:01:37.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:37.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:37.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:37.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:37.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:37.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:37.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:37.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:37.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:01:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:01:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:32 2026-02-22T20:01:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:01:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:37.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:01:37.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:33' 2026-02-22T20:01:37.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:01:37.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:01:37.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:37.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:01:37.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:37.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:37.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:37.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:37.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:37.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:37.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:37.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:32' 2026-02-22T20:01:37.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:01:32' 2026-02-22T20:01:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:32' 2026-02-22T20:01:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:37.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:37.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:01:32cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 20:01:33' 2026-02-22T20:01:37.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:37.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:37.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:32' 2026-02-22T20:01:37.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:37.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:01:37.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:37.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:01:37.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:37.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:37.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:37.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:01:37.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:37.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:37.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:37.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:01:37.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:01:38.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:38.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:39.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:01:39.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:39.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:01:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:01:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:01:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:01:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:01:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:39.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:01:39.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:39.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:01:39.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:39.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:01:39.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:39.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:39.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:39.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:39.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:39.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:01:39.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:01:39.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:39.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:01:39.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:01:39.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:01:39.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:39.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:01:39.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:01:39.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:39.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:01:43.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 95 2026-02-22T20:01:43.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:01:43.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:01:43.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:43.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:43.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:43.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:01:43.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:01:43.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:43.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:43.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:43.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:01:43.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:43.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:43.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:43.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:43.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:43.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:43.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:43.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:01:43.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:01:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:01:43.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=95 2026-02-22T20:01:43.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:01:43.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:43.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:01:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:01:44.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:01:44.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:01:44.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=95 2026-02-22T20:01:44.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 95 = 95 2026-02-22T20:01:44.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:01:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:01:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:45.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:43 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790497,"replay_state":"idle"} 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:37 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 95 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.be14b0ff-1130-444f-b36e-cae77078454e (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:01:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:01:45.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:01:45.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:45.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:01:45.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:45.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:01:45.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:45.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:01:45.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:45.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:45.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:45.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:45.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:43' 2026-02-22T20:01:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:01:43' 2026-02-22T20:01:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:43' 2026-02-22T20:01:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:45.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:01:43cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790497,"replay_state":"idle"}2026-02-22 20:01:37' 2026-02-22T20:01:45.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:45.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:45.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:43' 2026-02-22T20:01:45.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:45.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:01:45.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:01:45.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:01:45.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:01:45.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:45.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:45.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:01:45.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:45.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:45.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:45.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:45.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:45.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:45.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:45.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:01:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:46.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:01:46.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:46.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790497,"replay_state":"idle"} 2026-02-22T20:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:37 2026-02-22T20:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:43' 2026-02-22T20:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:01:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:01:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790497,"replay_state":"idle"} 2026-02-22T20:01:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:01:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:01:46.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:46.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:46.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:46.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:37' 2026-02-22T20:01:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:01:37' 2026-02-22T20:01:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:01:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:37' 2026-02-22T20:01:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:46.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790497,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:01:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:01:43' 2026-02-22T20:01:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:46.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:37' 2026-02-22T20:01:46.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:01:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:01:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:01:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:01:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:01:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:01:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:46.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:01:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:01:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:01:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:01:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:01:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:01:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:47.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:01:47.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:47.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:47.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:01:47.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:47.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:47.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:47.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:47.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:01:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:01:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:01:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:01:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:01:47.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:47.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:01:49.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:01:49.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:49.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:49.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:49.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:01:49.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:49.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:49.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:01:49.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:49.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:01:49.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:49.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:01:49.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:01:49.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:49.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:49.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:49.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:01:49.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:49.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:01:49.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:49.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:49.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:01:49.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:01:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:01:49.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:50.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:01:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:01:50.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:50.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:50.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:50.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:01:50.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:50.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:48 2026-02-22T20:01:50.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:50.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:01:50.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:50.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:01:50.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:48' 2026-02-22T20:01:50.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:01:50.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:01:50.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:50.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:01:51.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:51.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:51.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:51.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:51.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:51.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:51.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:51.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:48' 2026-02-22T20:01:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:01:48' 2026-02-22T20:01:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:48' 2026-02-22T20:01:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:51.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:51.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:51.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:01:48cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 20:01:48' 2026-02-22T20:01:51.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:51.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:51.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:48' 2026-02-22T20:01:51.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:51.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:01:51.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:51.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:01:51.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:51.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:51.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:51.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:01:51.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:51.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:51.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:51.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:51.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:51.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:51.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:51.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:52.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:01:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:01:52.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:01:52.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:01:52.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:52.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:01:52.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:01:52.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:48 2026-02-22T20:01:52.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:01:52.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:01:52.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:01:52.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:01:52.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:48' 2026-02-22T20:01:52.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:01:52.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:01:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:01:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:01:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:01:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:52.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:01:52.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:52.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:01:52.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:01:52.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:01:52.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:01:52.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:48' 2026-02-22T20:01:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:01:48' 2026-02-22T20:01:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:01:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:48' 2026-02-22T20:01:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:01:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:01:52.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:01:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 20:01:48cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:01:48' 2026-02-22T20:01:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:01:52.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:01:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:48' 2026-02-22T20:01:52.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:01:52.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:01:52.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:01:52.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:01:52.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:01:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:01:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:01:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:01:54.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:01:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:01:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:01:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:01:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:01:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:01:55.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:01:55.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:55.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:55.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:55.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:01:55.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:01:55.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:01:55.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:01:55.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:01:55.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:01:55.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:55.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:01:55.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:01:55.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:01:55.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:55.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:01:55.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:01:55.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:01:55.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:01:55.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:01:55.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:01:55.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:01:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:01:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:01:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:01:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:01:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:01:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:01:55.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:55.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:55.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:01:55.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:01:55.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:01:55.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:01:55.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:01:55.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:01:55.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:01:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:01:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:01:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:01:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:01:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:55.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:01:55.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:01:55.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:01:55.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:55.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:55.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:55.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:01:59.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 98 2026-02-22T20:01:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:01:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:01:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:59.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:01:59.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:01:59.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:59.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:59.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:01:59.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:01:59.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:01:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:01:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:01:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:01:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:01:59.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=98 2026-02-22T20:01:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:01:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:01:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:01:59.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:59.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:01:59.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:59.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:59.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:01:59.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:01:59.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:01:59.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:01:59.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=98 2026-02-22T20:01:59.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 98 = 98 2026-02-22T20:01:59.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:01:59.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:01:59.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:01:59.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:01:59.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:01:59.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:01:59.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:01:59.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:01:59.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:01:59.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:02:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:02:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:00.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:00.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:02:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:02:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:02:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:00.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:57 2026-02-22T20:02:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:02:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:02:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790513,"replay_state":"idle"} 2026-02-22T20:02:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:53 2026-02-22T20:02:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:02:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 98 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.3debb344-e184-4d44-a1a3-1948da134d84 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:02:00.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:02:00.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:02:00.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:00.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:02:00.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:00.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:02:00.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:00.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:02:00.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:00.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:00.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:00.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:00.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:57' 2026-02-22T20:02:00.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:01:57' 2026-02-22T20:02:00.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:00.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:00.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:00.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:02:00.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:57' 2026-02-22T20:02:00.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:00.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:00.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:01:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790513,"replay_state":"idle"}2026-02-22 20:01:53' 2026-02-22T20:02:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:00.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:57' 2026-02-22T20:02:00.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:02:00.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:02:00.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:02:00.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:02:00.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:00.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:00.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:02:00.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:00.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:00.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:00.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:00.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:00.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:00.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:00.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:02:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:01.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:01.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790513,"replay_state":"idle"} 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:53 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:01:57' 2026-02-22T20:02:01.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:02:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:02:01.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:01.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790513,"replay_state":"idle"} 2026-02-22T20:02:01.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:01.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:02:01.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:01.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:02:01.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:01.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:01.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:01.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:53' 2026-02-22T20:02:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:01:53' 2026-02-22T20:02:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:01:53' 2026-02-22T20:02:01.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:01.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:01.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:01.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790513,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:01:53cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:01:57' 2026-02-22T20:02:01.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:01.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:01.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:01:53' 2026-02-22T20:02:01.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:01.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:02:01.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:02:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:02:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:02:01.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:01.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:01.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:02:01.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:02:01.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:01.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:01.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:01.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:02:02.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:02.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:02:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:03.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:03.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:03.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:03.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:03.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:03.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:02:03.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:02:03.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:03.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:03.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:03.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:02:03.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:03.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:02:03.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:02:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:02:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:02:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:03.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:03.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:03.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:03.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:02:03.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:02:03.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:03.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:02:03.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:02:03.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:02:03.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:03.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:02:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:02:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:02:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:05.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:02:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:02:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:02:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:02:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:05.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:05.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:05.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:05.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:05.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:02:05.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:02:05.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:02:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:02:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:02:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:06.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:06.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:02:06.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:06.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:06.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:02:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:03 2026-02-22T20:02:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:02:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:02:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:02' 2026-02-22T20:02:06.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:02:06.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:02:06.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:06.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:02:06.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:06.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:06.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:06.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:06.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:06.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:06.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:06.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:06.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:03' 2026-02-22T20:02:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:02:03' 2026-02-22T20:02:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:03' 2026-02-22T20:02:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 20:02:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:02:02' 2026-02-22T20:02:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:06.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:06.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:03' 2026-02-22T20:02:06.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:06.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:02:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:06.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:06.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:06.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:02:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:07.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:02:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:02:07.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:02 2026-02-22T20:02:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:02:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:02:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:03' 2026-02-22T20:02:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:02:07.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:02:07.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:07.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:02:07.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:07.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:07.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:07.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:07.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:07.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:07.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:07.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:02' 2026-02-22T20:02:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:02:02' 2026-02-22T20:02:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:02' 2026-02-22T20:02:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:07.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:02:02cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 20:02:03' 2026-02-22T20:02:07.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:07.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:07.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:02' 2026-02-22T20:02:07.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:02:07.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:07.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:02:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:07.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:07.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:07.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:02:07.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:07.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:07.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:07.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:02:07.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:02:08.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:02:08.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:02:08.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:08.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:08.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:08.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:02:08.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:08.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:08.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:08.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:02:08.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:02:08.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:08.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:02:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:02:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:02:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:02:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:02:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:02:09.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:09.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:09.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:09.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:02:09.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:09.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:09.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:02:09.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:02:09.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:09.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:09.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:02:09.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:02:09.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:09.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:09.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:02:09.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:09.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:09.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:09.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:09.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:09.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:02:09.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:02:09.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:09.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:02:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:02:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:02:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:09.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:02:09.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:02:09.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:09.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:09.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:09.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:02:09.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:02:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:02:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:02:13.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 101 2026-02-22T20:02:13.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:02:13.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:02:13.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:13.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:13.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:13.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:02:13.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:02:13.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:02:13.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:02:13.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:02:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:02:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:13.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:02:13.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:02:13.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:02:13.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:02:13.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=101 2026-02-22T20:02:13.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:02:13.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:13.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:02:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:02:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:14.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:14.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:02:14.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:02:14.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:02:14.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:02:14.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=101 2026-02-22T20:02:14.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 101 = 101 2026-02-22T20:02:14.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:14.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:02:14.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:14.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:14.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:14.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:02:14.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:14.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:14.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:14.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:15.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:02:15.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:15.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:15.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:15.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:02:15.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:15.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:15.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:15.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:15.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:02:15.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:15.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:13 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790527,"replay_state":"idle"} 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:07 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 101 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.3e4961ae-dfa4-41a2-b603-4c4e0211979b (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:02:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:02:15.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:15.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:02:15.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:15.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:02:15.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:15.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:02:15.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:15.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:15.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:15.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:15.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:13' 2026-02-22T20:02:15.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:02:13' 2026-02-22T20:02:15.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:15.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:15.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:15.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:02:15.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:13' 2026-02-22T20:02:15.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:15.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:15.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:15.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:02:13cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790527,"replay_state":"idle"}2026-02-22 20:02:07' 2026-02-22T20:02:15.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:15.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:15.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:13' 2026-02-22T20:02:15.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:15.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:02:15.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:02:15.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:02:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:02:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:02:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:16.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:02:16.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:16.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:16.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:02:16.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:16.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:16.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:02:16.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790527,"replay_state":"idle"} 2026-02-22T20:02:16.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:16.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:07 2026-02-22T20:02:16.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:16.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:02:16.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:02:16.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:02:16.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:13' 2026-02-22T20:02:16.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:02:16.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:02:16.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:16.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790527,"replay_state":"idle"} 2026-02-22T20:02:16.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:16.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:02:16.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:16.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:02:16.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:16.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:16.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:16.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:07' 2026-02-22T20:02:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:02:07' 2026-02-22T20:02:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:16.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:07' 2026-02-22T20:02:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:16.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:16.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790527,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:02:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:02:13' 2026-02-22T20:02:16.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:16.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:07' 2026-02-22T20:02:16.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:16.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:02:16.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:02:16.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:02:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:02:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:16.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:16.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:02:16.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:16.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:16.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:16.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:02:16.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:02:16.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:02:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:02:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:02:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:02:16.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:16.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:17.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:02:17.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:17.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:17.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:17.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:02:17.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:17.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:17.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:02:17.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:17.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:02:17.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:17.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:02:17.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:02:17.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:17.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:17.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:17.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:02:17.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:17.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:17.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:17.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:02:17.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:02:17.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:17.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:17.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:02:17.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:17.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:17.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:17.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:17.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:02:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:02:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:17.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:02:17.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:02:17.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:02:17.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:17.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:02:19.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:02:19.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:19.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:19.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:19.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:02:19.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:19.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:02:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:02:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:02:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:02:19.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:19.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:19.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:02:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:19.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:19.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:19.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:19.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:02:19.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:02:19.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:19.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:19.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:02:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:20.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:02:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:02:20.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:20.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:02:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:02:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:20.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:21.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:02:21.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:21.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:21.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:21.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:21.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:21.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:21.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:21.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:21.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:02:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:02:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:18 2026-02-22T20:02:21.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:02:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:02:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:18' 2026-02-22T20:02:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:02:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:02:21.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:21.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:02:21.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:21.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:21.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:21.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:21.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:21.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:21.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:21.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:21.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:18' 2026-02-22T20:02:21.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:02:18' 2026-02-22T20:02:21.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:21.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:21.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:21.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:21.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:18' 2026-02-22T20:02:21.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:21.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:21.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:21.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:02:18cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 20:02:18' 2026-02-22T20:02:21.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:21.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:21.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:18' 2026-02-22T20:02:21.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:02:21.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:21.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:21.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:21.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:02:21.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:21.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:21.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:21.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:21.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:21.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:21.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:21.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:22.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:02:22.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:22.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:22.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:22.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:22.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:22.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:22.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:22.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:22.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:02:22.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:22.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:22.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:22.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:02:22.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:22.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:18 2026-02-22T20:02:22.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:02:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:02:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:18' 2026-02-22T20:02:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:02:22.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:02:22.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:22.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:02:22.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:22.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:22.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:22.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:22.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:22.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:22.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:22.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:22.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:18' 2026-02-22T20:02:22.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:02:18' 2026-02-22T20:02:22.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:22.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:22.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:22.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:22.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:18' 2026-02-22T20:02:22.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:22.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:22.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:22.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 20:02:18cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:02:18' 2026-02-22T20:02:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:22.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:22.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:18' 2026-02-22T20:02:22.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:22.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:02:22.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:22.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:02:22.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:22.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:22.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:22.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:02:22.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:22.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:22.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:22.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:02:22.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:02:24.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:02:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:02:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:02:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:02:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:02:24.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:24.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:02:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:02:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:02:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:02:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:25.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:25.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:25.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:25.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:25.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:25.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:25.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:25.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:02:25.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:02:25.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:25.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:02:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:02:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:02:25.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:02:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:02:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:02:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:02:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:02:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:25.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:02:29.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 104 2026-02-22T20:02:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:02:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:02:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:29.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:29.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:02:29.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:02:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:02:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:02:29.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:02:29.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:02:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:02:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:02:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:02:29.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:02:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=104 2026-02-22T20:02:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:02:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:02:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:02:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:02:29.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:02:29.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:02:29.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:02:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=104 2026-02-22T20:02:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 104 = 104 2026-02-22T20:02:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:02:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:29.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:02:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:29.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:02:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:02:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:30.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:27 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790543,"replay_state":"idle"} 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:23 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 104 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.0ed13e23-b632-423d-884e-5025a31cf8f0 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:02:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:02:30.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:02:30.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:02:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:02:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:02:30.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:30.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:30.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:30.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:30.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:27' 2026-02-22T20:02:30.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:02:27' 2026-02-22T20:02:30.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:30.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:30.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:30.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:02:30.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:27' 2026-02-22T20:02:30.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:30.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:30.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:02:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790543,"replay_state":"idle"}2026-02-22 20:02:23' 2026-02-22T20:02:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:30.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:27' 2026-02-22T20:02:30.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:30.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:02:30.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:02:30.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:02:30.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:02:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:30.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:02:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:30.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:02:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:31.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:31.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:31.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:02:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:31.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790543,"replay_state":"idle"} 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:23 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:27' 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:02:31.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:31.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790543,"replay_state":"idle"} 2026-02-22T20:02:31.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:02:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:31.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:02:31.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:31.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:31.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:23' 2026-02-22T20:02:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:02:23' 2026-02-22T20:02:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:31.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:31.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:23' 2026-02-22T20:02:31.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:31.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:31.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:31.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790543,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:02:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:02:27' 2026-02-22T20:02:31.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:31.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:31.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:23' 2026-02-22T20:02:31.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:31.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:02:31.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:02:31.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:02:31.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:02:31.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:31.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:31.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:02:31.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:02:31.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:31.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:31.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:31.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:02:32.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:02:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:02:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:02:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:02:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:02:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:32.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:02:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:02:33.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:33.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:33.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:02:33.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:02:33.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:33.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:33.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:02:33.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:02:33.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:33.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:33.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:33.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:33.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:33.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:33.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:02:33.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:02:33.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:33.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:33.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:02:33.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:33.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:33.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:33.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:33.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:02:33.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:02:33.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:02:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:02:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:02:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:02:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:02:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:02:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:35.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:02:35.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:35.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:02:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:02:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:02:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:02:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:02:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:02:35.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:02:35.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:35.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:35.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:02:35.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:35.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:02:35.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:02:35.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:02:35.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:35.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:35.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:02:35.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:35.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:35.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:02:35.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:35.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:35.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:02:35.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:02:35.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:02:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:02:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:35.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:02:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:02:36.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:36.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:36.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:36.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:02:36.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:36.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:33 2026-02-22T20:02:36.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:36.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:02:36.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:36.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:02:36.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:32' 2026-02-22T20:02:36.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:02:36.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:02:36.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:36.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:02:36.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:36.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:36.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:36.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:36.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:36.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:36.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:33' 2026-02-22T20:02:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:02:33' 2026-02-22T20:02:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:33' 2026-02-22T20:02:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:36.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 20:02:33cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:02:32' 2026-02-22T20:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:36.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:33' 2026-02-22T20:02:36.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:36.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:02:36.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:36.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:02:36.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:36.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:36.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:36.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:02:36.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:36.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:36.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:36.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:36.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:36.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:36.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:36.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:37.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:02:37.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:37.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:37.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:37.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:37.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:37.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:37.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:37.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:37.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:02:37.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:02:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:32 2026-02-22T20:02:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:02:37.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:02:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:33' 2026-02-22T20:02:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:02:37.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:02:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:37.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:02:37.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:37.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:37.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:37.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:37.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:37.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:37.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:37.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:37.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:32' 2026-02-22T20:02:37.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:02:32' 2026-02-22T20:02:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:32' 2026-02-22T20:02:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:37.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:37.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:02:32cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 20:02:33' 2026-02-22T20:02:37.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:37.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:37.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:32' 2026-02-22T20:02:37.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:37.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:02:37.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:37.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:02:37.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:37.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:37.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:37.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:02:37.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:37.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:37.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:37.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:02:37.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:02:39.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:40.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:02:40.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:40.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:40.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:40.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:02:40.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:40.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:02:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:02:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:02:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:02:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:40.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:02:40.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:40.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:40.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:02:40.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:02:40.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:40.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:02:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:02:40.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:40.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:40.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:02:40.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:40.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:40.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:40.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:40.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:40.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:02:40.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:02:40.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:40.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T20:02:40.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:40.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:02:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:02:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:02:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:42.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:02:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:02:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:02:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:02:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:42.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:02:42.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:02:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:02:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:02:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:02:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:42.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:02:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:02:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:02:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:02:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:42.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:02:42.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:42.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:42.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:42.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:42.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:42.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:02:42.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:02:42.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:42.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T20:02:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:42.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 4 2026-02-22T20:02:46.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:02:46.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:46.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:46.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:46.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:02:46.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:46.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:46.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:02:46.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:46.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:02:46.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:46.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:02:46.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:02:46.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:46.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:46.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:46.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:02:46.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:02:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:02:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:02:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:02:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:02:46.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:02:46.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:02:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:02:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:46.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:02:46.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:46.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:46.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:46.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:02:46.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:02:46.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:46.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:02:46.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:02:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:02:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:02:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:02:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:02:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:02:46.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:02:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:02:50.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 107 2026-02-22T20:02:50.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:02:50.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:02:50.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:50.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:50.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:50.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:02:50.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:02:50.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:50.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:02:50.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:02:50.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:02:50.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:02:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:02:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:02:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:50.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:02:50.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:02:50.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:02:50.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:02:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=107 2026-02-22T20:02:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:02:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:02:50.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:02:50.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:02:50.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:50.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:50.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:50.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:02:50.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:02:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:02:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:02:50.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=107 2026-02-22T20:02:50.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 107 = 107 2026-02-22T20:02:50.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:50.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:02:50.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:50.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:50.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:50.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:02:50.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:50.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:50.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:50.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:51.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:02:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:02:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:02:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:02:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:02:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:48 2026-02-22T20:02:51.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:51.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:02:51.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:02:51.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790558,"remote_snapshot_timestamp":1771790558,"replay_state":"idle"} 2026-02-22T20:02:51.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:47 2026-02-22T20:02:51.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:02:51.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 107 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.2aa27ea5-5313-42ab-b247-0b1890265050 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:02:51.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:02:51.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:02:51.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:51.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:02:51.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:51.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:02:51.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:02:51.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:51.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:51.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:51.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:48' 2026-02-22T20:02:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:02:48' 2026-02-22T20:02:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:02:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:48' 2026-02-22T20:02:51.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:51.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:51.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:02:48cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790558,"remote_snapshot_timestamp":1771790558,"replay_state":"idle"}2026-02-22 20:02:47' 2026-02-22T20:02:51.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:51.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:51.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:48' 2026-02-22T20:02:51.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:51.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:02:51.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:02:51.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:02:51.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:02:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:02:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:51.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:51.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:52.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:02:52.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:52.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:02:52.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:52.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:52.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:02:52.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790558,"remote_snapshot_timestamp":1771790558,"replay_state":"idle"} 2026-02-22T20:02:52.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:52.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:47 2026-02-22T20:02:52.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:52.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:02:52.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:02:52.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:02:52.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:48' 2026-02-22T20:02:52.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:02:52.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:02:52.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790558,"remote_snapshot_timestamp":1771790558,"replay_state":"idle"} 2026-02-22T20:02:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:02:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:02:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:52.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:52.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:47' 2026-02-22T20:02:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:02:47' 2026-02-22T20:02:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:02:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:47' 2026-02-22T20:02:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:52.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:52.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790558,"remote_snapshot_timestamp":1771790558,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:02:47cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:02:48' 2026-02-22T20:02:52.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:52.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:47' 2026-02-22T20:02:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:52.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:02:52.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:02:52.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:02:52.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:02:52.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:52.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:52.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:02:52.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:52.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:52.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:52.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:02:53.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:02:53.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:53.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:54.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:02:54.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:54.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:54.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:54.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:02:54.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:54.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:54.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:02:54.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:54.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:02:54.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:54.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:02:54.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:02:54.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:54.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:54.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:02:54.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:02:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:02:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:02:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:02:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:54.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:54.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:02:54.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:02:54.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:54.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:54.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:02:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:02:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:54.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:02:54.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:54.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:54.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:54.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:54.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:02:54.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:02:54.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:02:54.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:02:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:02:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:54.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:02:56.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:02:56.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:56.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:56.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:56.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:02:56.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:02:56.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:02:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:02:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:02:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:02:56.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:56.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:02:56.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:02:56.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:02:56.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:56.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:02:56.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:02:56.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:02:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:02:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:02:56.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:02:56.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:56.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:56.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:02:56.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:02:56.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:02:56.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:02:56.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:02:56.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:02:56.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:02:56.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:56.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:02:56.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:02:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:02:57.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:57.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:02:57.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:02:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:02:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:53 2026-02-22T20:02:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:02:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:02:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:55' 2026-02-22T20:02:57.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:02:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:02:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:02:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:02:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:59.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:59.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:59.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:02:59.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:02:59.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:02:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:02:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:02:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:58 2026-02-22T20:02:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:02:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:02:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:02:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:02:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:57' 2026-02-22T20:02:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:02:59.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:02:59.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:02:59.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:02:59.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:02:59.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:59.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:02:59.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:59.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:02:59.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:02:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:02:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:02:59.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:58' 2026-02-22T20:02:59.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:02:58' 2026-02-22T20:02:59.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:02:59.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:59.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:59.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:59.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:58' 2026-02-22T20:02:59.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:02:59.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:02:59.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:02:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:02:58cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:02:57' 2026-02-22T20:02:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:02:59.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:02:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:58' 2026-02-22T20:02:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:02:59.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:02:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:02:59.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:02:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:02:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:59.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:02:59.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:02:59.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:02:59.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:02:59.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:02:59.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:02:59.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:02:59.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:02:59.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:02:59.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:00.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:03:00.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:00.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:00.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:00.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:00.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:00.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:00.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:00.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:00.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:03:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:57 2026-02-22T20:03:01.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:01.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:03:01.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:01.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:01.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:02:58' 2026-02-22T20:03:01.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:03:01.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:03:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:01.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:01.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:57' 2026-02-22T20:03:01.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:02:57' 2026-02-22T20:03:01.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:01.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:01.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:01.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:01.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:02:57' 2026-02-22T20:03:01.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:01.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:01.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:02:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:02:58' 2026-02-22T20:03:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:01.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:01.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:02:57' 2026-02-22T20:03:01.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:03:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:03:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:01.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:01.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:01.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:03:01.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:01.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:01.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:01.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:03:01.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:03:03.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:03:03.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:03:03.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:03.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:03.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:03:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:03:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:03:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:03.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:04.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:03:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:03:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:03:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:03:04.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:03:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:03:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:04.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:03:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:03:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:04.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:04.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:03:04.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:04.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:04.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:04.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:04.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:04.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:04.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:03:04.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:03:04.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:04.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:03:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 110 2026-02-22T20:03:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:03:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:03:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:08.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:08.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:08.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:08.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:03:08.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:08.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:03:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:03:08.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:08.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:08.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:08.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:03:08.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:03:08.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:03:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=110 2026-02-22T20:03:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:03:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:03:08.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:08.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:08.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:08.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:08.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:08.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:03:08.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:03:08.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=110 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 110 = 110 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:03:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:03:09.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:09.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:09.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:03:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:03:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:07 2026-02-22T20:03:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:03:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:03:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790582,"replay_state":"idle"} 2026-02-22T20:03:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:03 2026-02-22T20:03:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:03:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 110 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.2dc3ba29-cc04-44dd-947f-2bad13787158 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:03:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:03:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:03:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:09.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:03:09.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:09.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:03:09.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:09.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:03:09.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:09.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:09.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:09.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:09.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:07' 2026-02-22T20:03:09.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:03:07' 2026-02-22T20:03:09.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:09.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:09.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:09.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:09.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:07' 2026-02-22T20:03:09.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:09.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:09.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:09.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:03:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790582,"replay_state":"idle"}2026-02-22 20:03:03' 2026-02-22T20:03:09.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:09.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:07' 2026-02-22T20:03:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:09.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:03:09.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:03:09.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:03:09.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:03:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:03:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:09.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:03:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:03:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:03:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790582,"replay_state":"idle"} 2026-02-22T20:03:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:03 2026-02-22T20:03:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:03:10.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:03:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:03:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:07' 2026-02-22T20:03:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:03:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:03:10.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790582,"replay_state":"idle"} 2026-02-22T20:03:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:03:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:10.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:03:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:03' 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:03:03' 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:03' 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790582,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:03:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:03:07' 2026-02-22T20:03:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:10.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:10.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:03' 2026-02-22T20:03:10.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:10.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:03:10.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:03:10.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:03:10.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:03:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:03:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:03:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:10.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:03:11.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:03:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:03:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:03:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:11.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:11.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:03:11.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:03:11.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:11.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:12.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:03:12.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:12.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:12.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:12.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:03:12.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:12.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:03:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:03:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:03:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:03:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:03:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:03:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:12.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:12.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:12.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:12.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:12.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:12.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:03:12.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:03:12.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:03:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:03:14.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:03:14.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:14.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:14.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:14.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:03:14.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:14.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:14.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:03:14.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:14.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:03:14.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:14.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:03:14.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:03:14.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:14.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:14.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:03:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:14.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:14.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:14.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:03:14.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:03:14.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:14.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:14.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:14.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:14.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:03:14.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:03:14.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:03:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:03:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:14.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:14.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:14.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:15.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:03:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:15.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:13 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:12' 2026-02-22T20:03:15.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:03:15.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:03:15.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:15.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:03:15.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:15.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:15.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:15.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:15.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:15.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:13' 2026-02-22T20:03:15.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:03:13' 2026-02-22T20:03:15.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:15.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:15.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:15.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:15.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:13' 2026-02-22T20:03:15.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:15.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:15.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:15.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:03:13cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:03:12' 2026-02-22T20:03:15.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:15.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:13' 2026-02-22T20:03:15.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:03:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:03:15.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:15.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:03:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:16.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:03:16.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:16.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:16.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:16.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:16.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:16.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:16.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:16.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:16.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:03:16.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:16.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:16.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:16.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:16.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:16.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:12 2026-02-22T20:03:16.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:16.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:03:16.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:16.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:16.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:13' 2026-02-22T20:03:16.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:03:16.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:03:16.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:16.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:03:16.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:16.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:16.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:16.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:16.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:16.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:16.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:16.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:16.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:12' 2026-02-22T20:03:16.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:03:12' 2026-02-22T20:03:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:12' 2026-02-22T20:03:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:16.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:16.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:03:12cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:03:13' 2026-02-22T20:03:16.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:16.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:12' 2026-02-22T20:03:16.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:16.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:03:16.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:03:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:16.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:16.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:16.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:03:16.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:16.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:16.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:16.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:03:16.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:03:18.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:03:18.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:03:18.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:18.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:18.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:18.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:03:18.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:18.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:18.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:18.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:03:18.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:03:18.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:18.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:19.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:03:19.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:19.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:19.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:19.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:03:19.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:19.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:19.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:03:19.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:19.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:03:19.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:19.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:03:19.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:03:19.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:19.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:19.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:19.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:03:19.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:19.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:19.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:03:19.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:03:19.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:19.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:19.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:03:19.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:03:19.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:19.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:19.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:03:19.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:19.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:19.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:19.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:19.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:19.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:03:19.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:03:19.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:19.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:03:19.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:03:19.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:03:19.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:19.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:03:19.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:03:19.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:19.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:03:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:03:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:03:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:03:23.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 113 2026-02-22T20:03:23.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:03:23.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:03:23.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:23.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:23.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:23.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:03:23.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:03:23.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:23.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:23.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:23.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:03:23.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:23.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:23.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:23.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:23.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:23.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:23.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:23.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:03:23.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:03:23.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:03:23.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=113 2026-02-22T20:03:23.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:03:23.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:23.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:03:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:03:23.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:03:23.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:03:23.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=113 2026-02-22T20:03:23.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 113 = 113 2026-02-22T20:03:23.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:23.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:03:23.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:23.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:23.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:23.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:23.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:23.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:23.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:23.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:24.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:03:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:24.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:03:24.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:24.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:24.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:03:24.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:03:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:23 2026-02-22T20:03:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:03:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:03:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790597,"replay_state":"idle"} 2026-02-22T20:03:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:17 2026-02-22T20:03:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:03:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 113 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.646c3462-2119-4dea-a152-02d4b59306ec (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:03:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:03:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:03:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:03:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:03:24.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:24.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:03:24.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:23' 2026-02-22T20:03:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:03:23' 2026-02-22T20:03:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:23' 2026-02-22T20:03:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:24.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:03:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790597,"replay_state":"idle"}2026-02-22 20:03:17' 2026-02-22T20:03:24.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:24.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:24.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:23' 2026-02-22T20:03:24.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:24.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:03:24.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:03:24.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:03:24.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:03:24.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:24.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:24.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:03:24.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:24.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:24.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:24.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:24.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:24.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:24.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:24.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:25.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:03:25.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:25.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:03:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:03:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790597,"replay_state":"idle"} 2026-02-22T20:03:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:25.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:17 2026-02-22T20:03:25.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:25.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:03:25.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:03:25.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:03:25.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:23' 2026-02-22T20:03:25.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:03:25.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:03:25.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:25.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790597,"replay_state":"idle"} 2026-02-22T20:03:25.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:25.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:03:25.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:25.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:03:25.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:25.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:25.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:25.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:17' 2026-02-22T20:03:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:03:17' 2026-02-22T20:03:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:17' 2026-02-22T20:03:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790597,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:03:17cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:03:23' 2026-02-22T20:03:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:17' 2026-02-22T20:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:26.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:03:26.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:03:26.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:03:26.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:03:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:03:26.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:26.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:26.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:26.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:03:26.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:27.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:03:27.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:27.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:27.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:27.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:03:27.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:03:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:03:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:03:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:03:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:27.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:03:27.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:27.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:27.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:03:27.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:03:27.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:27.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:27.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:03:27.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:03:27.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:03:27.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:27.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:03:27.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:03:27.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:27.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:27.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:27.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:27.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:27.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:03:27.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:03:27.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:27.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:27.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:03:27.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:27.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:27.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:27.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:27.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:27.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:03:27.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:03:27.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:27.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:03:27.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:03:27.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:03:27.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:27.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:03:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:03:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:29.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:03:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:29.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:03:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:03:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:29.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:03:29.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:03:29.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:29.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:29.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:29.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:03:29.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:03:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:29.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:29.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:29.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:03:29.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:03:29.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:29.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:29.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:03:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:29.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:29.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:29.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:03:29.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:03:29.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:03:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:03:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:29.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:03:30.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:30.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:30.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:30.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:30.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:30.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:30.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:30.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:30.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:03:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:30.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:28 2026-02-22T20:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:28' 2026-02-22T20:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:03:30.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:03:30.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:30.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:03:30.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:30.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:30.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:30.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:30.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:28' 2026-02-22T20:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:03:28' 2026-02-22T20:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:28' 2026-02-22T20:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:03:28cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 20:03:28' 2026-02-22T20:03:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:30.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:28' 2026-02-22T20:03:30.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:30.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:03:30.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:30.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:03:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:30.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:30.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:03:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:30.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:03:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:31.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:31.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:31.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:31.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:03:31.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:31.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:31.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:31.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:03:31.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:31.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:28 2026-02-22T20:03:31.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:31.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:03:31.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:31.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:31.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:28' 2026-02-22T20:03:31.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:03:31.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:03:31.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:31.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:03:31.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:31.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:31.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:31.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:31.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:31.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:31.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:31.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:31.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:28' 2026-02-22T20:03:31.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:03:28' 2026-02-22T20:03:31.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:31.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:31.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:31.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:31.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:28' 2026-02-22T20:03:31.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:31.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:31.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:31.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 20:03:28cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:03:28' 2026-02-22T20:03:31.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:31.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:31.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:28' 2026-02-22T20:03:31.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:31.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:03:31.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:31.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:03:31.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:31.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:31.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:31.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:03:31.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:31.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:31.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:31.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:03:31.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:03:33.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:03:33.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:03:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:03:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:03:33.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:03:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:03:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:34.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:34.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:34.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:34.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:34.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:03:34.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:03:34.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:34.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:34.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:03:38.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 116 2026-02-22T20:03:38.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:03:38.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:03:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:03:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:38.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:03:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:03:38.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:38.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:38.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:38.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:38.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:38.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:38.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:38.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:03:38.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:03:38.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:03:38.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=116 2026-02-22T20:03:38.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:03:38.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:38.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:03:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:03:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:03:38.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:03:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=116 2026-02-22T20:03:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 116 = 116 2026-02-22T20:03:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:03:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:38.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:38.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:38.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:39.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:03:39.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:39.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:39.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:39.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:39.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:39.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:39.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:39.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:39.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:03:39.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:03:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:03:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:37 2026-02-22T20:03:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:03:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:03:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790612,"replay_state":"idle"} 2026-02-22T20:03:39.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:33 2026-02-22T20:03:39.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:03:39.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 116 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.c8278d2b-00f0-44cc-be40-b4f01b186e34 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:03:39.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:03:39.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:03:39.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:39.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:03:39.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:39.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:03:39.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:39.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:03:39.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:39.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:39.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:39.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:39.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:37' 2026-02-22T20:03:39.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:03:37' 2026-02-22T20:03:39.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:39.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:39.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:39.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:39.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:37' 2026-02-22T20:03:39.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:39.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:39.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:39.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:03:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790612,"replay_state":"idle"}2026-02-22 20:03:33' 2026-02-22T20:03:39.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:39.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:39.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:37' 2026-02-22T20:03:39.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:03:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:03:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:03:39.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:03:39.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:39.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:39.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:03:39.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:39.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:39.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:39.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:39.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:39.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:39.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:39.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:40.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:03:40.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:40.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:40.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:40.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:40.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:40.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:40.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:40.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:40.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:03:40.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:40.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:40.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:03:40.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790612,"replay_state":"idle"} 2026-02-22T20:03:40.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:40.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:33 2026-02-22T20:03:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:03:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:03:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:03:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:37' 2026-02-22T20:03:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:03:40.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:03:40.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790612,"replay_state":"idle"} 2026-02-22T20:03:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:03:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:40.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:03:40.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:40.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:40.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:40.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:40.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:33' 2026-02-22T20:03:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:03:33' 2026-02-22T20:03:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:33' 2026-02-22T20:03:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:40.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790612,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:03:33cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:03:37' 2026-02-22T20:03:40.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:40.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:40.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:33' 2026-02-22T20:03:40.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:40.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:03:40.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:03:40.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:03:40.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:03:40.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:40.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:40.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:03:40.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:03:40.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:40.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:40.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:40.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:03:41.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:03:41.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:03:41.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:41.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:41.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:41.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:03:41.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:41.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:41.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:41.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:03:41.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:03:41.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:41.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:42.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:03:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:03:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:03:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:03:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:03:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:03:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:42.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:42.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:42.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:42.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:03:42.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:42.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:42.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:03:42.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:03:42.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:42.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:42.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:42.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:42.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:42.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:42.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:42.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:42.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:03:42.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:03:42.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:42.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:03:42.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:03:42.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:03:42.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:42.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:03:44.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:03:44.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:44.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:44.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:44.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:03:44.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:44.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:44.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:03:44.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:03:44.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:44.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:44.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:03:44.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:03:44.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:44.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:44.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:44.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:03:44.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:44.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:44.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:03:44.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:03:44.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:44.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:03:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:03:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:03:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:03:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:44.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:44.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:03:44.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:03:44.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:44.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:03:44.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:44.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:03:44.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:44.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:44.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:44.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:44.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:44.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:44.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:44.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:45.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:03:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:45.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:03:45.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:45.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:45.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:45.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:45.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:45.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:43 2026-02-22T20:03:45.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:45.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:03:45.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:45.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:45.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:42' 2026-02-22T20:03:45.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:03:45.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:03:45.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:45.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:03:45.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:45.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:45.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:45.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:45.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:45.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:45.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:45.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:43' 2026-02-22T20:03:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:03:43' 2026-02-22T20:03:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:43' 2026-02-22T20:03:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:45.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:03:43cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:03:42' 2026-02-22T20:03:45.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:45.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:45.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:43' 2026-02-22T20:03:45.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:03:45.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:45.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:03:45.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:03:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:46.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:03:46.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:46.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:46.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:46.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:46.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:46.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:46.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:46.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:46.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:03:46.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:46.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:46.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:46.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:46.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:46.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:42 2026-02-22T20:03:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:03:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:43' 2026-02-22T20:03:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:03:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:03:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:03:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:46.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:42' 2026-02-22T20:03:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:03:42' 2026-02-22T20:03:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:46.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:42' 2026-02-22T20:03:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:46.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:46.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:03:42cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:03:43' 2026-02-22T20:03:46.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:46.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:46.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:42' 2026-02-22T20:03:46.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:46.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:03:46.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:46.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:03:46.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:46.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:46.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:46.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:03:46.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:46.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:46.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:46.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:03:46.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:03:47.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:03:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:03:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:03:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:03:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:03:47.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:47.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:48.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:03:48.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:48.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:48.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:48.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:03:48.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:48.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:48.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:03:48.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:48.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:03:48.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:48.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:03:48.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:03:48.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:48.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:48.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:48.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:03:48.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:48.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:48.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:03:48.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:48.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:48.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:48.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:03:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:03:48.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:48.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:03:52.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 119 2026-02-22T20:03:52.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:03:52.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:03:52.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:52.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:52.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:52.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:03:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:03:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:52.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:52.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:03:52.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:52.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:03:52.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:03:52.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:03:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:03:52.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=119 2026-02-22T20:03:52.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:03:52.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:03:52.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:03:52.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:52.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:52.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:52.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:52.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:03:52.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:03:52.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:03:52.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:03:52.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=119 2026-02-22T20:03:52.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 119 = 119 2026-02-22T20:03:52.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:52.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:03:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:52.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:53.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:03:53.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:53.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:53.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:53.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:53.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:53.884 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:53.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:53.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:53.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:03:53.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:53.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:53.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:03:53.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:03:53.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:53.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:53 2026-02-22T20:03:53.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:53.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:03:53.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:03:53.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790626,"replay_state":"idle"} 2026-02-22T20:03:53.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:47 2026-02-22T20:03:53.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:03:53.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 119 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.bb6afe35-01fb-4865-b528-d96577a6cc97 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:03:53.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:03:53.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:03:53.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:53.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:03:53.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:53.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:03:53.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:53.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:03:53.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:53.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:53.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:53.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:53' 2026-02-22T20:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:03:53' 2026-02-22T20:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:53' 2026-02-22T20:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:53.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:53.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:03:53cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790626,"replay_state":"idle"}2026-02-22 20:03:47' 2026-02-22T20:03:53.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:53' 2026-02-22T20:03:53.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:53.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:03:53.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:03:53.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:03:53.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:03:53.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:53.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:53.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:03:53.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:53.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:53.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:53.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:53.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:53.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:53.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:53.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:03:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:54.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:54.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:03:55.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:55.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:55.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:03:55.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790626,"replay_state":"idle"} 2026-02-22T20:03:55.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:55.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:47 2026-02-22T20:03:55.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:55.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:03:55.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:03:55.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:03:55.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:53' 2026-02-22T20:03:55.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:03:55.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:03:55.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:55.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790626,"replay_state":"idle"} 2026-02-22T20:03:55.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:55.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:03:55.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:55.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:03:55.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:55.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:55.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:55.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:47' 2026-02-22T20:03:55.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:03:47' 2026-02-22T20:03:55.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:55.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:55.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:55.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:03:55.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:47' 2026-02-22T20:03:55.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:55.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:55.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:55.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790626,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:03:47cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:03:53' 2026-02-22T20:03:55.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:55.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:55.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:47' 2026-02-22T20:03:55.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:55.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:03:55.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:03:55.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:03:55.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:03:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:03:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:55.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:03:55.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:03:55.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:03:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:03:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:03:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:03:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:55.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:56.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:03:56.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:56.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:56.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:56.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:03:56.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:56.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:56.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:03:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:03:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:03:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:03:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:03:56.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:56.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:56.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:03:56.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:03:56.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:56.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:56.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:03:56.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:03:56.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:56.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:56.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:03:56.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:56.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:56.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:03:56.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:56.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:56.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:56.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:03:56.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:03:56.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:03:56.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:03:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:03:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:03:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:03:56.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:56.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:03:56.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:56.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:56.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:56.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:56.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:56.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:03:56.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:03:56.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:56.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:03:56.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:03:56.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:03:56.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:56.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:03:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:03:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:03:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:03:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:03:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:03:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:03:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:03:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:03:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:03:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:03:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:03:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:03:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:03:58.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:03:58.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:03:58.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:03:58.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:03:58.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:03:58.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:03:58.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:03:58.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:03:58.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:03:58.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:03:58.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:58.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:58.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:03:58.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:03:58.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:03:58.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:03:58.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:03:58.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:03:58.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:03:58.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:03:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:03:59.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:03:59.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:59.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:59.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:59.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:59.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:59.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:59.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:03:59.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:03:59.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:03:59.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:59.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:59.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:03:59.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:58 2026-02-22T20:03:59.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:03:59.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:03:59.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:03:59.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:03:59.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:57' 2026-02-22T20:03:59.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:03:59.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:03:59.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:03:59.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:03:59.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:03:59.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:59.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:03:59.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:59.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:03:59.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:03:59.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:03:59.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:03:59.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:58' 2026-02-22T20:03:59.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:03:58' 2026-02-22T20:03:59.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:03:59.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:59.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:59.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:59.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:58' 2026-02-22T20:03:59.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:03:59.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:03:59.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:03:59.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:03:58cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:03:57' 2026-02-22T20:03:59.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:03:59.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:03:59.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:58' 2026-02-22T20:03:59.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:03:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:03:59.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:03:59.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:03:59.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:03:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:03:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:03:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:03:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:03:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:03:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:03:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:03:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:03:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:04:00.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:04:00.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:00.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:00.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:00.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:04:00.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:00.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:57 2026-02-22T20:04:00.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:00.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:04:00.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:00.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:04:00.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:03:58' 2026-02-22T20:04:00.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:04:00.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:04:00.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:04:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:00.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:00.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:00.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:00.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:00.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:00.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:57' 2026-02-22T20:04:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:03:57' 2026-02-22T20:04:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:03:57' 2026-02-22T20:04:00.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:00.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:00.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:00.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:03:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:03:58' 2026-02-22T20:04:00.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:00.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:00.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:03:57' 2026-02-22T20:04:00.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:00.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:04:00.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:00.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:04:00.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:04:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:04:00.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:04:02.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:04:02.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:04:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:04:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:04:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:04:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:02.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:03.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:04:03.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:03.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:03.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:03.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:04:03.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:03.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:04:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:04:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:04:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:04:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:03.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:03.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:03.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:03.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:03.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:03.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:03.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:03.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:03.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:03.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:03.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:04:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:04:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:03.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T20:04:03.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:03.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:04:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:04:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:05.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:04:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:05.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:04:05.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:05.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:04:05.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:05.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:04:05.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:04:05.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:05.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:05.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:05.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:05.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:05.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:05.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:05.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:05.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:05.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:05.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:05.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:05.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:05.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:05.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:05.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:05.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:05.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:04:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:04:05.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:05.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:04:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:04:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:04:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:04:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:04:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:04:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:04:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:04:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:04:09.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 122 2026-02-22T20:04:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:04:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:04:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:09.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:09.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:04:09.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:09.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:09.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:04:09.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:04:09.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:09.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:04:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:04:09.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:04:09.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=122 2026-02-22T20:04:09.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:04:09.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:09.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:04:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:04:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:04:09.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:04:09.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=122 2026-02-22T20:04:09.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 122 = 122 2026-02-22T20:04:09.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:09.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:04:09.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:09.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:09.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:09.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:09.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:09.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:09.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:09.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:10.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:04:10.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:10.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:10.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:10.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:10.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:10.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:10.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:10.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:10.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:04:10.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:10.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:10.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:04:10.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:04:10.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:10.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:07 2026-02-22T20:04:10.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:10.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:04:10.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:04:10.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790641,"replay_state":"idle"} 2026-02-22T20:04:10.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:03 2026-02-22T20:04:10.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:04:10.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 122 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.9cbcf0ae-373c-4b1f-8710-206b6d34f652 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:04:10.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:04:10.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:04:10.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:10.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:04:10.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:10.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:04:10.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:10.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:04:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:07' 2026-02-22T20:04:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:04:07' 2026-02-22T20:04:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:07' 2026-02-22T20:04:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:10.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:04:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790641,"replay_state":"idle"}2026-02-22 20:04:03' 2026-02-22T20:04:10.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:10.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:07' 2026-02-22T20:04:10.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:04:10.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:04:10.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:04:10.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:11.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:04:11.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:11.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:11.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:11.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:11.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:11.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:11.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:11.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:11.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:04:11.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:11.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:11.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:04:11.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790641,"replay_state":"idle"} 2026-02-22T20:04:11.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:11.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:03 2026-02-22T20:04:11.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:11.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:04:11.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:04:11.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:04:11.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:07' 2026-02-22T20:04:11.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:04:11.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:04:11.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:11.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790641,"replay_state":"idle"} 2026-02-22T20:04:11.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:11.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:04:11.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:11.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:04:11.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:11.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:11.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:11.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:11.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:03' 2026-02-22T20:04:11.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:04:03' 2026-02-22T20:04:11.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:11.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:11.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:11.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:11.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:03' 2026-02-22T20:04:11.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:11.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:11.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:11.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790641,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:04:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:04:07' 2026-02-22T20:04:11.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:11.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:11.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:03' 2026-02-22T20:04:11.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:04:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:04:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:04:11.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:04:11.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:11.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:11.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:04:11.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:04:11.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:11.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:11.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:11.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:04:12.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:04:12.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:04:12.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:12.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:12.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:04:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:04:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:04:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:12.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:13.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:04:13.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:13.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:04:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:04:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:04:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:04:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:04:13.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:13.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:13.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:04:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:04:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:13.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:13.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:13.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:13.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:13.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:13.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:04:13.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:04:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:13.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:04:13.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:04:13.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:04:13.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:13.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:04:15.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:04:15.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:15.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:15.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:15.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:04:15.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:15.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:15.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:04:15.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:15.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:04:15.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:15.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:04:15.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:04:15.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:15.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:15.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:15.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:15.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:15.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:15.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:15.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:15.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:15.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:16.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:04:16.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:16.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:16.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:16.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:04:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:16.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:04:16.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:16.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:13 2026-02-22T20:04:16.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:16.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:04:16.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:16.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:04:16.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:12' 2026-02-22T20:04:16.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:04:16.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:04:16.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:16.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:04:16.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:16.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:16.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:16.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:13' 2026-02-22T20:04:16.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:04:13' 2026-02-22T20:04:16.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:16.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:16.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:16.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:16.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:13' 2026-02-22T20:04:16.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:16.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:16.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:16.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 20:04:13cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:04:12' 2026-02-22T20:04:16.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:16.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:16.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:13' 2026-02-22T20:04:16.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:16.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:04:16.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:16.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:04:16.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:16.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:16.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:16.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:04:16.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:16.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:16.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:16.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:16.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:16.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:16.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:16.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:17.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:04:17.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:17.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:17.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:17.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:17.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:17.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:17.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:17.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:17.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:04:17.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:04:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:12 2026-02-22T20:04:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:04:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:04:17.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:13' 2026-02-22T20:04:17.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:04:17.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:04:17.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:17.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:04:17.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:17.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:17.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:17.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:12' 2026-02-22T20:04:17.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:04:12' 2026-02-22T20:04:17.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:17.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:17.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:17.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:17.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:12' 2026-02-22T20:04:17.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:17.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:17.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:04:12cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 20:04:13' 2026-02-22T20:04:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:17.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:17.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:12' 2026-02-22T20:04:17.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:17.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:04:17.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:17.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:04:17.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:17.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:17.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:17.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:04:17.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:17.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:17.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:17.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:04:17.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:04:18.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:04:18.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:04:18.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:18.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:18.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:18.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:04:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:04:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:04:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:18.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:04:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:04:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:19.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:04:19.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:19.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:04:19.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:19.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:04:19.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:04:19.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:19.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:19.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:19.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:04:19.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:04:19.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:19.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:19.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:19.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:19.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:19.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:04:19.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:04:19.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:19.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:19.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:04:23.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 125 2026-02-22T20:04:23.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:04:23.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:04:23.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:23.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:23.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:23.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:04:23.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:04:23.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:23.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:23.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:23.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:04:23.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:23.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:23.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:23.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:23.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:23.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:23.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:23.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:04:23.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:04:23.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:04:23.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=125 2026-02-22T20:04:23.700 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:04:23.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:23.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:04:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:04:23.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:04:23.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:04:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=125 2026-02-22T20:04:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 125 = 125 2026-02-22T20:04:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:04:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:24.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:04:24.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:24.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:24.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:24.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:24.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:24.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:24.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:24.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:24.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:04:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:04:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:04:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:23 2026-02-22T20:04:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:04:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:04:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790657,"replay_state":"idle"} 2026-02-22T20:04:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:17 2026-02-22T20:04:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:04:24.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 125 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.c2c5ee34-6b3e-4bf9-9be2-d11fccb21244 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:04:24.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:04:24.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:04:24.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:24.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:04:24.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:24.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:04:24.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:24.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:04:24.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:24.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:24.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:24.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:24.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:23' 2026-02-22T20:04:24.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:04:23' 2026-02-22T20:04:24.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:24.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:24.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:24.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:24.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:23' 2026-02-22T20:04:24.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:24.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:24.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:25.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:04:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790657,"replay_state":"idle"}2026-02-22 20:04:17' 2026-02-22T20:04:25.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:25.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:25.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:23' 2026-02-22T20:04:25.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:25.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:04:25.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:04:25.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:04:25.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:04:25.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:25.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:25.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:04:25.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:25.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:25.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:25.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:25.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:25.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:25.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:25.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:04:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:26.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:26.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:04:26.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:04:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790657,"replay_state":"idle"} 2026-02-22T20:04:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:17 2026-02-22T20:04:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:04:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:04:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:04:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:23' 2026-02-22T20:04:26.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:04:26.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:04:26.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:26.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790657,"replay_state":"idle"} 2026-02-22T20:04:26.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:04:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:26.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:04:26.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:26.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:26.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:26.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:17' 2026-02-22T20:04:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:04:17' 2026-02-22T20:04:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:26.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:17' 2026-02-22T20:04:26.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:26.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:26.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:26.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790657,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:04:17cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:04:23' 2026-02-22T20:04:26.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:26.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:26.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:17' 2026-02-22T20:04:26.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:04:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:04:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:04:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:04:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:04:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:04:26.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:04:26.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:04:26.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:26.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:26.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:26.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:04:26.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:26.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:26.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:26.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:04:26.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:04:26.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:26.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:27.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:04:27.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:27.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:27.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:27.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:27.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:27.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:04:27.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:27.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:27.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:04:27.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:04:27.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:27.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:27.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:04:27.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:04:27.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:27.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:27.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:04:27.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:27.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:27.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:04:27.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:27.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:27.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:27.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:04:27.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:04:27.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:04:27.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:04:27.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:27.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:27.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:27.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:27.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:27.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:04:27.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:04:27.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:27.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:04:27.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:04:27.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:04:27.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:27.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:04:29.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:04:29.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:29.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:29.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:29.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:04:29.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:29.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:29.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:04:29.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:29.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:04:29.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:29.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:04:29.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:04:29.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:29.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:29.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:29.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:04:29.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:29.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:04:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:04:29.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:29.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:29.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:04:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:29.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:29.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:29.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:04:29.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:04:29.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:04:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:04:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:30.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:04:30.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:30.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:04:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:04:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:28 2026-02-22T20:04:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:04:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:04:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:28' 2026-02-22T20:04:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:04:30.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:04:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:30.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:04:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:30.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:30.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:30.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:30.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:28' 2026-02-22T20:04:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:04:28' 2026-02-22T20:04:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:28' 2026-02-22T20:04:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:30.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:04:28cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 20:04:28' 2026-02-22T20:04:30.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:30.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:30.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:28' 2026-02-22T20:04:30.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:30.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:04:30.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:30.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:04:30.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:30.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:30.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:04:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:30.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:31.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:04:31.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:31.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:31.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:31.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:31.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:31.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:31.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:04:31.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:31.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:04:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:28 2026-02-22T20:04:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:04:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:04:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:28' 2026-02-22T20:04:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:04:31.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:04:31.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:31.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:04:31.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:31.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:31.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:31.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:31.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:31.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:31.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:28' 2026-02-22T20:04:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:04:28' 2026-02-22T20:04:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:28' 2026-02-22T20:04:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:31.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:32.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 20:04:28cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:04:28' 2026-02-22T20:04:32.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:32.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:32.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:28' 2026-02-22T20:04:32.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:04:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:04:32.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:32.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:32.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:32.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:04:32.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:32.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:32.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:32.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:04:32.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:04:33.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:04:33.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:04:33.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:33.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:33.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:33.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:04:33.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:33.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:33.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:33.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:04:33.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:04:33.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:33.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:34.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:04:34.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:34.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:34.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:34.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:04:34.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:34.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:34.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:04:34.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:04:34.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:34.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:34.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:04:34.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:04:34.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:34.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:34.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:34.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:34.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:34.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:04:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:04:34.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:04:34.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:34.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:34.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:34.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:34.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:34.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:04:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:04:34.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:34.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:04:34.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:04:34.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:04:34.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:34.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:04:34.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:04:34.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:04:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:04:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:04:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:04:38.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 128 2026-02-22T20:04:38.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:04:38.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:04:38.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:38.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:38.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:38.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:38.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:04:38.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:38.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:04:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:04:38.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:04:38.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:04:38.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:04:38.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=128 2026-02-22T20:04:38.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:04:38.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:38.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:04:38.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:38.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:04:38.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:04:38.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:04:38.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=128 2026-02-22T20:04:38.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 128 = 128 2026-02-22T20:04:38.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:38.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:04:38.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:38.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:38.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:38.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:38.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:38.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:38.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:38.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:39.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:04:39.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:39.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:39.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:39.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:39.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:39.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:39.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:39.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:39.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:04:39.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:39.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:39.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:04:39.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:04:39.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:39.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:37 2026-02-22T20:04:39.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:39.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:04:39.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:04:39.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790672,"replay_state":"idle"} 2026-02-22T20:04:39.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:33 2026-02-22T20:04:39.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:04:39.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 128 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.63b97c5d-5a9c-4ecd-abe8-6e713d52a96b (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:04:39.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:04:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:04:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:39.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:04:39.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:39.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:04:39.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:39.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:04:39.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:39.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:39.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:39.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:39.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:37' 2026-02-22T20:04:39.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:04:37' 2026-02-22T20:04:39.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:39.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:39.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:39.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:39.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:37' 2026-02-22T20:04:39.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:39.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:39.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:39.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:04:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790672,"replay_state":"idle"}2026-02-22 20:04:33' 2026-02-22T20:04:39.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:39.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:39.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:37' 2026-02-22T20:04:39.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:39.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:04:39.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:04:39.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:04:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:04:39.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:39.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:39.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:04:39.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:39.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:39.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:39.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:39.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:39.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:39.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:39.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:40.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:04:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:04:40.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:40.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:04:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790672,"replay_state":"idle"} 2026-02-22T20:04:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:33 2026-02-22T20:04:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:04:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:04:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:04:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:37' 2026-02-22T20:04:40.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:04:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:04:40.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:40.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790672,"replay_state":"idle"} 2026-02-22T20:04:40.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:40.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:04:40.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:04:40.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:40.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:40.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:33' 2026-02-22T20:04:40.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:04:33' 2026-02-22T20:04:40.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:40.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:40.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:40.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:40.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:33' 2026-02-22T20:04:40.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:40.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:40.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:40.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790672,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:04:33cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:04:37' 2026-02-22T20:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:40.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:40.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:33' 2026-02-22T20:04:40.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:40.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:04:40.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:04:40.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:04:40.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:04:40.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:40.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:40.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:04:40.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:04:40.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:40.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:40.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:40.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:04:41.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:04:41.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:04:41.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:41.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:41.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:41.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:04:41.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:41.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:41.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:41.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:04:41.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:04:41.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:41.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:04:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:04:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:42.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:04:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:04:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:04:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:04:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:42.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:42.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:42.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:42.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:42.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:42.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:42.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:42.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:42.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:42.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:42.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:42.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:42.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:42.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:42.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:42.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:04:42.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:04:42.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:42.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:04:42.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:04:42.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:04:42.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:42.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:04:44.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:04:44.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:44.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:44.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:44.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:04:44.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:44.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:04:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:04:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:04:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:04:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:04:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:44.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:44.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:04:44.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:44.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:44.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:04:44.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:44.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:44.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:04:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:04:44.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:04:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:04:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:44.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:44.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:44.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:44.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:04:45.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:45.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:45.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:45.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:45.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:45.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:45.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:45.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:45.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:43 2026-02-22T20:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:04:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:42' 2026-02-22T20:04:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:04:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:04:45.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:45.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:45.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:45.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:45.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:45.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:45.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:43' 2026-02-22T20:04:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:04:43' 2026-02-22T20:04:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:45.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:45.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:43' 2026-02-22T20:04:45.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:45.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:45.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:45.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:04:43cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:04:42' 2026-02-22T20:04:45.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:45.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:45.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:43' 2026-02-22T20:04:45.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:45.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:04:45.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:45.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:04:45.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:04:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:45.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:45.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:45.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:45.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:46.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:04:46.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:46.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:04:46.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:04:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:42 2026-02-22T20:04:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:04:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:04:46.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:04:46.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:43' 2026-02-22T20:04:46.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:04:46.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:04:46.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:46.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:04:46.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:46.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:46.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:46.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:46.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:46.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:46.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:46.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:42' 2026-02-22T20:04:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:04:42' 2026-02-22T20:04:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:42' 2026-02-22T20:04:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:46.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:46.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:04:42cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:04:43' 2026-02-22T20:04:46.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:46.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:46.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:42' 2026-02-22T20:04:46.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:46.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:04:46.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:04:46.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:04:46.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:04:46.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:46.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:46.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:04:46.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:46.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:46.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:46.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:04:46.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:04:47.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:04:47.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:04:47.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:47.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:47.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:47.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:04:47.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:47.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:47.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:47.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:04:47.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:04:47.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:47.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:48.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:04:48.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:48.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:04:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:04:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:04:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:04:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:04:48.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:04:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:04:48.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:04:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:04:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:04:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:04:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:04:48.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:48.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:48.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:48.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:04:48.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:04:48.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:48.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:48.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:04:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:48.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:48.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:04:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:04:48.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:48.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:04:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:04:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:04:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:04:48.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:04:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:04:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:04:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:04:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:48.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:04:52.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 131 2026-02-22T20:04:52.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:04:52.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:04:52.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:52.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:52.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:52.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:04:52.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:04:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:04:52.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:04:52.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:04:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:04:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:04:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:04:52.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:04:52.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=131 2026-02-22T20:04:52.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:04:52.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:04:52.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:04:52.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:04:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:04:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:04:52.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:04:52.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=131 2026-02-22T20:04:52.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 131 = 131 2026-02-22T20:04:52.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:52.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:04:52.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:52.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:52.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:52.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:52.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:52.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:52.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:52.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:53.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:04:53.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:53.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:53.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:53.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:53.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:53.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:53.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:53.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:53.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:04:53.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:53.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:53.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:04:53.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:04:53.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:53.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:53 2026-02-22T20:04:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:04:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:04:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790686,"replay_state":"idle"} 2026-02-22T20:04:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:47 2026-02-22T20:04:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:04:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 131 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.8e1edb8d-09c3-4710-9117-aee97d0fa0f9 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:04:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:04:54.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:04:54.000 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:54.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:04:54.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:54.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:04:54.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:54.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:04:54.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:54.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:54.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:53' 2026-02-22T20:04:54.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:04:53' 2026-02-22T20:04:54.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:54.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:54.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:54.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:04:54.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:53' 2026-02-22T20:04:54.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:54.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:54.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:04:53cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790686,"replay_state":"idle"}2026-02-22 20:04:47' 2026-02-22T20:04:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:54.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:53' 2026-02-22T20:04:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:54.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:04:54.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:04:54.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:04:54.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:04:54.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:54.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:54.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:04:54.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:54.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:54.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:54.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:54.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:54.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:54.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:54.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:55.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:04:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:55.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:04:55.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:04:55.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:04:55.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:04:55.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790686,"replay_state":"idle"} 2026-02-22T20:04:55.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:04:55.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:47 2026-02-22T20:04:55.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:04:55.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:04:55.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:04:55.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:04:55.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:53' 2026-02-22T20:04:55.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:04:55.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:04:55.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:04:55.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790686,"replay_state":"idle"} 2026-02-22T20:04:55.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:04:55.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:04:55.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:04:55.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:04:55.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:04:55.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:04:55.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:04:55.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:04:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:47' 2026-02-22T20:04:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:04:47' 2026-02-22T20:04:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:04:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:47' 2026-02-22T20:04:55.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:55.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:55.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:04:55.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790686,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:04:47cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:04:53' 2026-02-22T20:04:55.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:04:55.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:04:55.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:47' 2026-02-22T20:04:55.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:04:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:04:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:04:55.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:04:55.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:04:55.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:04:55.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:04:55.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:55.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:55.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:55.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:04:55.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:55.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:55.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:55.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:04:55.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:04:55.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:55.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:56.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:04:56.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:56.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:56.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:56.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:04:56.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:56.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:56.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:04:56.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:04:56.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:56.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:56.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:04:56.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:04:56.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:56.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:56.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:56.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:04:56.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:04:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:04:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:04:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:04:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:04:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:56.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:56.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:56.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:56.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:04:56.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:04:56.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:56.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:56.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:04:56.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:56.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:56.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:56.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:56.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:56.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:04:56.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:04:56.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:56.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:04:56.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:04:56.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:04:56.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:56.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:04:58.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:04:58.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:04:58.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:58.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:58.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:04:58.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:04:58.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:04:58.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:04:58.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:04:58.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:04:58.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:58.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:04:58.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:04:58.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:04:58.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:58.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:04:58.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:04:58.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:04:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:04:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:04:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:04:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:04:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:04:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:04:58.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:04:58.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:04:58.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:04:58.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:04:58.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:58.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:58.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:04:58.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:04:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:04:58.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:04:58.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:04:58.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:04:58.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:04:58.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:04:58.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:04:58.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:58.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:58.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:58.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:58.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:58.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:58.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:04:58.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:04:59.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:04:59.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:04:59.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:04:59.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:04:59.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:04:59.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:04:59.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:04:59.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:04:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:04:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:05:00.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:00.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:00.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:00.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:05:00.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:00.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:58 2026-02-22T20:05:00.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:00.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:05:00.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:00.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:05:00.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:57' 2026-02-22T20:05:00.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:05:00.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:05:00.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:00.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:05:00.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:00.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:00.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:00.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:00.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:00.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:00.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:00.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:00.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:58' 2026-02-22T20:05:00.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:04:58' 2026-02-22T20:05:00.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:00.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:00.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:58' 2026-02-22T20:05:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:00.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:04:58cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:04:57' 2026-02-22T20:05:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:00.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:00.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:58' 2026-02-22T20:05:00.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:00.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:05:00.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:00.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:05:00.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:00.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:00.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:00.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:05:00.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:00.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:00.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:00.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:00.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:00.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:00.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:00.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:01.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:05:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:05:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:01.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:57 2026-02-22T20:05:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:05:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:05:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:04:58' 2026-02-22T20:05:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:05:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:05:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:01.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:05:01.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:01.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:01.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:01.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:01.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:01.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:01.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:01.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:01.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:57' 2026-02-22T20:05:01.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:04:57' 2026-02-22T20:05:01.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:01.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:01.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:01.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:01.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:04:57' 2026-02-22T20:05:01.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:01.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:01.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:01.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:04:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:04:58' 2026-02-22T20:05:01.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:01.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:01.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:04:57' 2026-02-22T20:05:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:01.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:05:01.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:01.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:05:01.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:05:01.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:05:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:05:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:05:02.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:05:02.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:02.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:02.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:02.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:05:02.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:02.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:02.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:02.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:05:02.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:05:02.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:02.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:03.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:05:03.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:03.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:03.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:03.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:05:03.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:03.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:03.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:05:03.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:03.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:05:03.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:03.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:05:03.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:05:03.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:03.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:03.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:03.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:05:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:05:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:05:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:05:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:05:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:05:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:03.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:03.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:03.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:03.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:03.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:03.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:03.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:05:03.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:05:03.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:03.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T20:05:03.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:03.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:05:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:05:05.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:05.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:05.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:05.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:05.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:05:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:05:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:05:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:05.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:05.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:05.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:05.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:05.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:05:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:05:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:05:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:05:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:05:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:05:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:05:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:05:09.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 134 2026-02-22T20:05:09.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:05:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:05:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:05:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:05:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:05:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:09.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:09.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:09.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:09.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:09.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:09.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:09.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:05:09.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:05:09.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:05:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=134 2026-02-22T20:05:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:05:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:05:09.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:05:09.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=134 2026-02-22T20:05:09.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 134 = 134 2026-02-22T20:05:09.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:09.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:05:09.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:09.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:09.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:09.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:09.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:09.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:09.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:09.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:10.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:05:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:10.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:05:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:05:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:05:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:07 2026-02-22T20:05:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:05:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:05:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790701,"replay_state":"idle"} 2026-02-22T20:05:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:03 2026-02-22T20:05:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:05:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 134 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.d62ba269-eeb9-47c3-8693-b21f76ab9eee (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:05:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:05:10.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:05:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:10.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:05:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:05:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:05:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:10.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:10.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:07' 2026-02-22T20:05:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:05:07' 2026-02-22T20:05:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:10.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:10.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:07' 2026-02-22T20:05:10.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:10.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:10.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:05:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790701,"replay_state":"idle"}2026-02-22 20:05:03' 2026-02-22T20:05:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:10.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:10.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:07' 2026-02-22T20:05:10.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:10.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:05:10.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:05:10.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:05:10.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:05:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:10.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:05:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:10.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:11.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:05:11.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:11.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:11.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:11.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:11.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:11.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:11.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:11.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:11.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:05:11.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:11.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:11.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:05:11.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790701,"replay_state":"idle"} 2026-02-22T20:05:11.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:11.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:03 2026-02-22T20:05:11.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:11.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:05:11.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:05:11.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:05:11.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:07' 2026-02-22T20:05:11.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:05:11.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:05:11.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:11.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790701,"replay_state":"idle"} 2026-02-22T20:05:11.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:11.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:05:11.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:11.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:05:11.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:11.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:11.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:11.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:11.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:03' 2026-02-22T20:05:11.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:05:03' 2026-02-22T20:05:11.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:11.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:11.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:11.746 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:11.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:03' 2026-02-22T20:05:11.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:11.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:11.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:11.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790701,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:05:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:05:07' 2026-02-22T20:05:11.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:11.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:11.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:03' 2026-02-22T20:05:11.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:11.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:05:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:05:11.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:05:11.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:05:11.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:11.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:11.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:05:11.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:05:11.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:11.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:11.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:11.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:05:12.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:05:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:05:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:05:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:05:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:05:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:12.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:05:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:13.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:05:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:13.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:05:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:05:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:05:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:05:13.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:13.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:05:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:05:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:05:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:05:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:05:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:05:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:13.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:13.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:13.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:13.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:05:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:05:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:13.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:05:13.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:05:13.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:05:13.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:13.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:05:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:05:15.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:15.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:15.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:15.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:05:15.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:15.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:15.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:05:15.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:15.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:05:15.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:15.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:05:15.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:05:15.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:15.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:15.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:15.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:15.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:15.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:05:15.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:05:15.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:15.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:15.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:15.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:05:15.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:15.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:15.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:15.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:15.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:15.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:15.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:15.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:15.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:05:15.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:05:15.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:05:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:05:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:16.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:05:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:16.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:05:16.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:05:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:13 2026-02-22T20:05:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:05:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:05:16.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:12' 2026-02-22T20:05:16.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:05:16.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:05:16.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:05:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:16.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:16.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:16.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:16.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:16.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:16.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:16.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:13' 2026-02-22T20:05:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:05:13' 2026-02-22T20:05:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:13' 2026-02-22T20:05:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:16.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:16.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:16.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 20:05:13cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:05:12' 2026-02-22T20:05:16.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:16.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:16.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:13' 2026-02-22T20:05:16.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:16.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:05:16.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:16.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:05:16.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:16.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:16.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:16.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:05:16.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:16.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:16.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:16.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:16.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:16.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:16.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:16.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:17.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:05:17.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:17.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:17.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:17.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:17.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:17.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:17.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:17.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:17.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:05:17.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:17.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:17.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:17.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:05:17.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:17.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:12 2026-02-22T20:05:17.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:17.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:05:17.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:17.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:05:17.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:13' 2026-02-22T20:05:17.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:05:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:05:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:17.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:05:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:17.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:17.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:17.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:17.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:17.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:12' 2026-02-22T20:05:17.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:05:12' 2026-02-22T20:05:17.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:17.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:17.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:17.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:17.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:12' 2026-02-22T20:05:17.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:17.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:17.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:17.468 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:05:12cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 20:05:13' 2026-02-22T20:05:17.468 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:17.468 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:17.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:12' 2026-02-22T20:05:17.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:17.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:05:17.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:17.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:05:17.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:17.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:17.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:17.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:05:17.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:17.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:17.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:17.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:05:17.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:05:18.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:05:18.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:05:18.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:18.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:18.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:18.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:05:18.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:18.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:18.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:18.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:05:18.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:05:18.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:18.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:19.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:05:19.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:19.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:19.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:19.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:05:19.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:19.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:19.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:05:19.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:19.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:05:19.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:19.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:05:19.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:05:19.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:19.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:19.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:19.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:05:19.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:19.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:19.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:05:19.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:05:19.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:19.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:19.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:05:19.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:05:19.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:19.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:19.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:05:19.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:19.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:19.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:19.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:19.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:19.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:05:19.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:05:19.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:19.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:05:19.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:05:19.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:05:19.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:19.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:05:19.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:05:19.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:19.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:19.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:19.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:05:19.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:05:19.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:05:19.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:19.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:19.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:19.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:05:23.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 137 2026-02-22T20:05:23.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:05:23.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:05:23.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:23.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:23.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:23.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:05:23.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:05:23.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:23.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:23.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:23.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:05:23.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:05:23.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:05:23.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:05:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=137 2026-02-22T20:05:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:05:23.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:23.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:05:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:05:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:05:24.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:05:24.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=137 2026-02-22T20:05:24.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 137 = 137 2026-02-22T20:05:24.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:24.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:05:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:25.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:05:25.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:25.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:25.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:25.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:25.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:25.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:25.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:25.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:25.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:05:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:25.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:25.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:05:25.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:05:25.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:25.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:23 2026-02-22T20:05:25.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:25.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:05:25.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:05:25.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790717,"replay_state":"idle"} 2026-02-22T20:05:25.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:17 2026-02-22T20:05:25.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:05:25.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 137 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.0897b23b-974f-4f5e-a028-3e1a97545675 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:05:25.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:05:25.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:05:25.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:25.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:05:25.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:25.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:05:25.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:25.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:05:25.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:25.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:25.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:25.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:25.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:23' 2026-02-22T20:05:25.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:05:23' 2026-02-22T20:05:25.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:25.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:25.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:25.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:25.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:23' 2026-02-22T20:05:25.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:25.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:25.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:25.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:05:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790717,"replay_state":"idle"}2026-02-22 20:05:17' 2026-02-22T20:05:25.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:25.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:25.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:23' 2026-02-22T20:05:25.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:25.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:05:25.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:05:25.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:05:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:05:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:05:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:26.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:05:26.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:26.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:26.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:26.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:26.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:26.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:26.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:26.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:26.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:05:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:05:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790717,"replay_state":"idle"} 2026-02-22T20:05:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:17 2026-02-22T20:05:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:26.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:05:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:05:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:05:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:23' 2026-02-22T20:05:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:05:26.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:05:26.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:26.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790717,"replay_state":"idle"} 2026-02-22T20:05:26.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:26.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:05:26.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:26.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:05:26.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:26.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:26.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:26.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:26.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:17' 2026-02-22T20:05:26.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:05:17' 2026-02-22T20:05:26.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:26.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:26.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:26.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:26.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:17' 2026-02-22T20:05:26.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:26.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:26.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:26.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790717,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:05:17cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:05:23' 2026-02-22T20:05:26.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:26.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:26.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:17' 2026-02-22T20:05:26.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:26.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:05:26.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:05:26.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:05:26.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:05:26.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:26.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:26.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:05:26.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:26.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:26.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:26.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:05:26.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:05:26.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:05:26.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:26.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:26.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:26.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:05:26.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:26.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:26.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:26.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:05:26.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:05:26.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:26.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:27.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:05:27.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:27.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:27.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:27.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:05:27.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:27.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:27.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:05:27.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:27.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:05:27.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:27.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:05:27.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:05:27.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:27.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:27.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:27.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:05:27.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:27.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:27.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:05:27.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:05:27.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:27.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:27.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:05:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:05:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:05:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:05:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:27.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:05:27.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:27.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:27.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:05:27.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:27.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:27.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:27.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:27.822 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:27.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:05:27.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:05:27.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:27.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:05:27.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:05:27.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:05:27.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:27.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:05:29.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:05:29.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:29.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:29.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:29.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:05:29.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:29.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:29.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:05:29.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:29.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:05:29.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:29.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:05:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:05:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:05:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:05:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:05:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:05:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:05:29.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:05:29.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:29.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:05:29.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:05:29.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:29.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:05:29.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:29.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:05:29.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:29.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:29.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:29.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:29.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:29.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:29.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:29.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:30.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:05:30.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:30.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:30.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:30.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:30.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:30.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:30.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:30.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:30.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:28 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:28' 2026-02-22T20:05:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:05:30.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:05:30.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:30.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:05:30.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:30.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:30.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:30.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:30.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:30.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:30.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:30.955 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:30.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:28' 2026-02-22T20:05:30.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:05:28' 2026-02-22T20:05:30.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:30.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:30.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:30.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:30.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:28' 2026-02-22T20:05:30.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:30.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:30.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:05:28cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 20:05:28' 2026-02-22T20:05:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:30.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:28' 2026-02-22T20:05:30.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:31.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:05:31.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:31.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:05:31.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:05:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:31.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:32.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:05:32.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:32.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:32.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:32.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:32.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:32.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:32.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:32.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:32.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:05:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:32.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:05:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:28 2026-02-22T20:05:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:05:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:05:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:28' 2026-02-22T20:05:32.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:05:32.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:05:32.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:32.041 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:05:32.041 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:32.041 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:32.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:32.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:32.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:28' 2026-02-22T20:05:32.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:05:28' 2026-02-22T20:05:32.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:32.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:28' 2026-02-22T20:05:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:32.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:32.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 20:05:28cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:05:28' 2026-02-22T20:05:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:32.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:28' 2026-02-22T20:05:32.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:32.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:05:32.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:32.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:05:32.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:32.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:05:34.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:34.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:35.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:05:35.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:35.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:35.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:35.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:05:35.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:35.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:35.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:05:35.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:35.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:05:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:05:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:05:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:35.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:35.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:35.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:05:35.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:05:35.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:35.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:35.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:35.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:05:35.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:35.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:35.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:35.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:05:35.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:35.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:35.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:35.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:35.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:35.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:05:35.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:05:35.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:35.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:35.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:35.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:35.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:35.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:35.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:05:35.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:05:35.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:35.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:05:35.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:05:35.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:05:35.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:05:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:05:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:05:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:05:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:05:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:35.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:05:39.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 140 2026-02-22T20:05:39.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:05:39.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:05:39.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:39.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:39.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:39.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:39.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:05:39.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:39.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:39.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:05:39.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:05:39.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:05:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:05:39.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:05:39.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=140 2026-02-22T20:05:39.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:05:39.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:39.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:05:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:39.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:05:39.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:05:39.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:05:39.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=140 2026-02-22T20:05:39.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 140 = 140 2026-02-22T20:05:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:05:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:40.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:05:40.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:40.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:40.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:40.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:40.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:40.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:40.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:40.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:40.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:05:40.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:05:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:05:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:37 2026-02-22T20:05:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:05:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:05:40.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790733,"replay_state":"idle"} 2026-02-22T20:05:40.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:33 2026-02-22T20:05:40.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:05:40.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 140 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.1df21406-0440-4c34-9c66-425ca5148224 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:05:40.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:05:40.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:05:40.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:40.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:05:40.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:40.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:05:40.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:40.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:05:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:40.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:37' 2026-02-22T20:05:40.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:05:37' 2026-02-22T20:05:40.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:40.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:40.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:40.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:40.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:37' 2026-02-22T20:05:40.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:40.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:40.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:40.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:05:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790733,"replay_state":"idle"}2026-02-22 20:05:33' 2026-02-22T20:05:40.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:40.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:40.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:37' 2026-02-22T20:05:40.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:05:40.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:05:40.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:05:40.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:05:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:05:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:40.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:41.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:05:41.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:41.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:41.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:41.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:41.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:41.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:41.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:41.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:41.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:05:41.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:05:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790733,"replay_state":"idle"} 2026-02-22T20:05:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:33 2026-02-22T20:05:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:05:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:05:41.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:05:41.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:37' 2026-02-22T20:05:41.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:05:41.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:05:41.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:41.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790733,"replay_state":"idle"} 2026-02-22T20:05:41.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:41.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:05:41.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:41.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:05:41.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:41.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:41.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:33' 2026-02-22T20:05:41.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:05:33' 2026-02-22T20:05:41.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:41.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:41.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:41.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:41.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:33' 2026-02-22T20:05:41.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:41.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:41.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790733,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:05:33cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:05:37' 2026-02-22T20:05:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:33' 2026-02-22T20:05:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:41.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:05:41.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:05:41.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:05:41.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:05:41.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:41.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:41.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:05:41.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test 2026-02-22T20:05:41.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:41.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:41.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:41.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2026-02-22T20:05:42.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:05:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:05:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:05:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:05:42.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:05:42.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:42.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:43.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:05:43.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:43.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:43.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:43.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:05:43.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:43.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:43.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:05:43.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:05:43.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:43.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:43.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:05:43.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:05:43.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:43.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:43.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:43.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:43.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:43.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:43.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:05:43.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:05:43.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:43.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:43.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:43.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:05:43.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:43.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:43.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:05:43.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:43.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:43.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:05:43.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:43.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:43.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:43.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:43.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:05:43.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:05:43.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:43.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:43.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:05:43.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:43.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:43.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:43.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:43.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:43.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:05:43.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:05:43.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:43.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:43.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:43.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:43.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:43.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:43.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:43.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:43.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:05:43.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:05:43.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:43.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:05:43.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:05:43.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:05:43.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:43.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:05:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:05:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:45.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:05:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:05:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:05:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:45.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:05:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:05:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:45.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:05:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:05:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:05:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:45.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:05:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:05:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:05:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:05:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:45.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:05:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:05:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:05:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:05:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:05:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:05:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:05:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:05:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:05:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:45.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:05:45.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:45.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:45.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:05:45.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:45.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:45.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:05:45.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:05:45.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:45.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:05:45.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:45.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:05:45.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:45.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:45.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:45.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:45.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:45.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:45.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:45.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:46.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:05:46.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:46.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:46.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:46.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:46.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:46.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:46.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:46.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:46.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:05:46.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:46.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:46.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:46.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:05:46.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:46.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:43 2026-02-22T20:05:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:05:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:05:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:42' 2026-02-22T20:05:46.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:05:46.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:05:46.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:46.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:05:46.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:46.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:46.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:46.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:46.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:46.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:46.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:46.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:46.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:43' 2026-02-22T20:05:46.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:05:43' 2026-02-22T20:05:46.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:43' 2026-02-22T20:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:46.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted43384346mirror.0trial0172026-02-22 20:05:43cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:05:42' 2026-02-22T20:05:46.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:46.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:46.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:43' 2026-02-22T20:05:46.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:05:46.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:46.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:46.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:46.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:05:46.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:46.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:46.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:46.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:46.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:46.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:46.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:46.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:47.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:05:47.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:47.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:47.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:47.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:47.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:47.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:47.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:47.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:47.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:05:47.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:05:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:42 2026-02-22T20:05:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:05:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:05:47.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:05:47.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:43' 2026-02-22T20:05:47.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:05:47.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:05:47.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:47.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:05:47.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:47.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:47.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:47.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:47.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:47.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:42' 2026-02-22T20:05:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:05:42' 2026-02-22T20:05:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:05:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:42' 2026-02-22T20:05:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:47.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:47.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:47.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary44524459mirror.0trial0172026-02-22 20:05:42cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 20:05:43' 2026-02-22T20:05:47.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:47.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:47.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:42' 2026-02-22T20:05:47.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:47.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:05:47.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:05:47.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:05:47.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:05:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test 2026-02-22T20:05:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:05:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2026-02-22T20:05:48.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test started 2026-02-22T20:05:49.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:05:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:49.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:05:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:05:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:49.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:05:49.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:05:49.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:49.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:49.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:49.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:05:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:05:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:05:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:05:49.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:05:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:05:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:05:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:05:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:05:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:05:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:05:49.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:49.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:05:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:05:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:05:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:05:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:05:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2026-02-22T20:05:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:05:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:05:49.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:05:49.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster1 mirror test 2026-02-22T20:05:49.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:49.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:49.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:49.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/test 2026-02-22T20:05:53.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 143 2026-02-22T20:05:53.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster2 cluster1 mirror test 2026-02-22T20:05:53.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster2 2026-02-22T20:05:53.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:53.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:53.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:53.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:05:53.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:05:53.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:53.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:53.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:05:53.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:05:53.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:05:53.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:53.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:53.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:53.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:53.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:53.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:05:53.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:05:53.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:05:53.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:05:53.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=143 2026-02-22T20:05:53.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:05:53.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:05:53.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:05:54.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:54.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:54.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:54.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:54.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:05:54.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:05:54.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:05:54.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:05:54.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=143 2026-02-22T20:05:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 143 = 143 2026-02-22T20:05:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2026-02-22T20:05:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:55.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2026-02-22T20:05:55.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:55.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:55.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:55.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:05:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:05:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:05:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:55.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:53 2026-02-22T20:05:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:05:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:05:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790747,"replay_state":"idle"} 2026-02-22T20:05:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:47 2026-02-22T20:05:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:05:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 143 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.f87f1f7f-3a7d-49b6-81a8-09e3a0f26573 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:05:55.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:05:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:05:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:55.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:55.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:05:55.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:55.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:55.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:53' 2026-02-22T20:05:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2026-02-22 20:05:53' 2026-02-22T20:05:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:05:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:53' 2026-02-22T20:05:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:55.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:55.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary43384346mirror.0trial0172026-02-22 20:05:53cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790747,"replay_state":"idle"}2026-02-22 20:05:47' 2026-02-22T20:05:55.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:55.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:55.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:53' 2026-02-22T20:05:55.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:05:55.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:05:55.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:05:55.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:05:55.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:55.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:55.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2026-02-22T20:05:55.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:55.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2026-02-22T20:05:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:05:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:05:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:56.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:05:56.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:05:56.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:05:56.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:05:56.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790747,"replay_state":"idle"} 2026-02-22T20:05:56.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:05:56.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:47 2026-02-22T20:05:56.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:05:56.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:05:56.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:05:56.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:05:56.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:53' 2026-02-22T20:05:56.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:05:56.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:05:56.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:05:56.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790747,"replay_state":"idle"} 2026-02-22T20:05:56.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:05:56.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:05:56.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:05:56.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:05:56.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:05:56.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:05:56.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:05:56.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:47' 2026-02-22T20:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2026-02-22 20:05:47' 2026-02-22T20:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:56.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:56.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:05:56.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:47' 2026-02-22T20:05:56.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:05:56.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:05:56.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:05:56.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790747,"replay_state":"idle"}44524459mirror.0trial0172026-02-22 20:05:47cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:05:53' 2026-02-22T20:05:56.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:05:56.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:05:56.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:47' 2026-02-22T20:05:56.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:05:56.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:05:56.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:05:56.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:05:56.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:05:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:05:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror test 2026-02-22T20:05:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:05:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:56.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:56.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2026-02-22T20:05:56.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:05:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2026-02-22T20:05:56.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2026-02-22T20:05:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:05:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:05:56.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:56.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:05:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:05:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:05:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:57.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:57.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:05:57.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:57.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:05:57.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:57.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:05:57.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:05:57.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:57.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:57.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:05:57.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:57.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:57.862 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:05:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:05:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:05:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:05:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:05:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:57.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:05:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:05:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:05:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:05:57.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:05:57.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:05:57.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:05:57.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:05:57.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:05:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:05:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = started 2026-02-22T20:05:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:05:59.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test stopped 2026-02-22T20:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:05:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2026-02-22T20:05:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:05:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:05:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:59.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:05:59.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:05:59.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:05:59.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:05:59.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:05:59.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:05:59.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:05:59.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2026-02-22T20:05:59.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:05:59.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:05:59.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:05:59.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:05:59.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:05:59.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:59.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:59.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:05:59.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:06:00.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:06:00.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:06:00.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:06:00.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:06:00.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:06:00.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:00.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2026-02-22T20:06:00.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:00.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:00.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:00.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:06:00.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:00.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:00.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:00.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:01.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2026-02-22T20:06:01.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:01.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:01.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:01.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:06:01.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:01.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:01.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:01.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:01.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:06:01.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:06:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:06:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:06:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:06:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:06:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:58 2026-02-22T20:06:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:06:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:06:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:06:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:06:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:58' 2026-02-22T20:06:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:06:01.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:06:01.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:06:01.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:06:01.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:01.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:06:01.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:06:01.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:06:01.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:06:01.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:06:01.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:06:01.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:06:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:58' 2026-02-22T20:06:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2026-02-22 20:05:58' 2026-02-22T20:06:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:06:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:58' 2026-02-22T20:06:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:06:01.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image is not primary43384346mirror.0trial0172026-02-22 20:05:58cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image demoted2026-02-22 20:05:58' 2026-02-22T20:06:01.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:06:01.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:06:01.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:58' 2026-02-22T20:06:01.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:06:01.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:06:01.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:06:01.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:06:01.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:01.121 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2026-02-22T20:06:01.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:01.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:01.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:01.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:06:01.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:01.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:01.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:01.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:02.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2026-02-22T20:06:02.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:02.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:58 2026-02-22T20:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:06:02.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:05:58' 2026-02-22T20:06:02.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:06:02.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:06:02.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:06:02.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:06:02.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:02.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:06:02.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:06:02.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:06:02.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:06:02.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:06:02.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:06:02.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:06:02.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:58' 2026-02-22T20:06:02.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2026-02-22 20:05:58' 2026-02-22T20:06:02.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:02.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:06:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:05:58' 2026-02-22T20:06:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:02.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:06:02.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+unknownremote image demoted44524459mirror.0trial0172026-02-22 20:05:58cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:05:58' 2026-02-22T20:06:02.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:06:02.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:06:02.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:05:58' 2026-02-22T20:06:02.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:06:02.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:06:02.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:06:02.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:06:02.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:06:02.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:02.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:02.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster2 mirror test 2026-02-22T20:06:02.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:02.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:02.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:02.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:06:02.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2026-02-22T20:06:04.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:06:04.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:06:04.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:04.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:04.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:04.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:06:04.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:04.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:04.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:04.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:06:04.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:04.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:04.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:05.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:06:05.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:05.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:06:05.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:06:05.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:05.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:06:05.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:05.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:06:05.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:05.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:06:05.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:06:05.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:05.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:05.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:05.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:06:05.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:05.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:05.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:05.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:06:05.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:05.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:05.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:05.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:06:05.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:05.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:05.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:06:05.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:06:05.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:05.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:06:05.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:06:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:06:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:06:05.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:06:05.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:06:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:06:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:06:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2026-02-22T20:06:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:06:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:05.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:06:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:06:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-22T20:06:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:05.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-22T20:06:09.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 146 2026-02-22T20:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2026-02-22T20:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.status 2026-02-22T20:06:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.status 2026-02-22T20:06:09.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:09.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:06:09.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.status 2026-02-22T20:06:09.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.status 2026-02-22T20:06:09.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:06:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:06:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.status 2026-02-22T20:06:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-22T20:06:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:09.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:06:09.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=146 2026-02-22T20:06:09.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:06:09.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:09.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:06:09.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:06:09.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:09.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:09.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:09.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.status 2026-02-22T20:06:09.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-22T20:06:09.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:09.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:06:09.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=146 2026-02-22T20:06:09.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 146 = 146 2026-02-22T20:06:09.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:09.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2026-02-22T20:06:09.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:09.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:09.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:09.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:09.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:09.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:09.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:09.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:10.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2026-02-22T20:06:10.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:10.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2026-02-22T20:06:10.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:06:10.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:06:10.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:06:10.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:06:10.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:06:10.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:07 2026-02-22T20:06:10.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:06:10.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:06:10.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:06:10.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790763,"replay_state":"idle"} 2026-02-22T20:06:10.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:03 2026-02-22T20:06:10.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:06:10.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 146 .mirror.primary.3958e471-d54e-4d1a-84d5-607022ebcf0e.832dce0d-bd0c-44e3-933b-638788f7dd56 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:06:10.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:06:10.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:06:10.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:06:10.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:06:10.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:10.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:06:10.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:06:10.468 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:06:10.468 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:06:10.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:06:10.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:06:10.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:06:10.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:07' 2026-02-22T20:06:10.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2026-02-22 20:06:07' 2026-02-22T20:06:10.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:10.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:10.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:10.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:10.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:06:07' 2026-02-22T20:06:10.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:10.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:10.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:06:10.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:06:07cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790763,"replay_state":"idle"}2026-02-22 20:06:03' 2026-02-22T20:06:10.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:06:10.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:06:10.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:07' 2026-02-22T20:06:10.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:06:10.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:06:10.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:06:10.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:06:10.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:10.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:10.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:06:11.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:11.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:11.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:11.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:11.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:11.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:11.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:11.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:11.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:06:11.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:06:11.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3958e471-d54e-4d1a-84d5-607022ebcf0e 2026-02-22T20:06:11.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:06:11.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790763,"replay_state":"idle"} 2026-02-22T20:06:11.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:06:11.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:03 2026-02-22T20:06:11.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:06:11.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:06:11.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:06:11.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:06:11.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:07' 2026-02-22T20:06:11.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:06:11.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:06:11.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:06:11.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790763,"replay_state":"idle"} 2026-02-22T20:06:11.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:11.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:06:11.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:06:11.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:06:11.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:06:11.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:06:11.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:06:11.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:06:11.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:03' 2026-02-22T20:06:11.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:06:03' 2026-02-22T20:06:11.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:11.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:11.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:06:11.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:11.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:06:03' 2026-02-22T20:06:11.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:11.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:11.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:06:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790763,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:06:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:06:07' 2026-02-22T20:06:11.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:06:11.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:06:11.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:03' 2026-02-22T20:06:11.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:06:11.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:06:11.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:06:11.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:06:11.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:06:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:06:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_mirror_snaps cluster1 mirror test 2026-02-22T20:06:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:06:11.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:06:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 snap ls mirror/test --all 2026-02-22T20:06:11.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c -F ' mirror (' 2026-02-22T20:06:11.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 3 -le 3 2026-02-22T20:06:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_mirror_snaps cluster2 mirror test 2026-02-22T20:06:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:06:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:06:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:06:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 snap ls mirror/test --all 2026-02-22T20:06:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c -F ' mirror (' 2026-02-22T20:06:11.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 1 -le 3 2026-02-22T20:06:11.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: force promote' 2026-02-22T20:06:11.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:06:11.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:06:11.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:06:11 TEST: force promote 2026-02-22T20:06:11.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:06:11 TEST: force promote 2026-02-22T20:06:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ force_promote_image=test_force_promote 2026-02-22T20:06:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test_force_promote snapshot 2026-02-22T20:06:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test_force_promote 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test_force_promote 2026-02-22T20:06:11.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:06:11.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:11.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:06:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test_force_promote snapshot 2026-02-22T20:06:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:06:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test_force_promote snapshot 2026-02-22T20:06:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:06:12.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test_force_promote 2026-02-22T20:06:12.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_force_promote': 2026-02-22T20:06:12.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:06:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:06:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:06:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 1500eb26b558 2026-02-22T20:06:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.1500eb26b558 2026-02-22T20:06:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:06:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:06:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:06:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:06:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:06:11 2026 2026-02-22T20:06:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:06:11 2026 2026-02-22T20:06:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:06:11 2026 2026-02-22T20:06:12.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 1500eb26b558 2026-02-22T20:06:12.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:06:12.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:06:12.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 789b3e4e-b7a0-4613-8861-7cdd7d2076e6 2026-02-22T20:06:12.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:06:12.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_force_promote 100 2026-02-22T20:06:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:06:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:06:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:06:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:06:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:06:12.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:06:12.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:06:12.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.27 bytes/sec: 24 MiB/s 2026-02-22T20:06:12.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote 2026-02-22T20:06:12.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:12.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:12.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:12.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped 2026-02-22T20:06:12.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:12.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:12.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:12.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:06:12.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:12.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:12.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:13.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:13.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:06:13.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:06:13.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:13.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:06:13.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:06:13.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:06:13.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:06:13.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:06:13.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-22T20:06:13.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:13.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:13.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:13.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:13.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:06:13.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:13.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:13.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:06:13.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:06:13.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:13.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:13.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:06:13.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:06:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:06:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:06:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:13.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:06:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:06:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:06:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:06:13.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:06:13.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:06:13.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:06:13.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:06:13.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:06:13.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:06:13.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:06:13.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:13.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-22T20:06:13.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:13.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:13.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:2 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:2 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:06:13.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:2 cluster2 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:2 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:2 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:13.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:06:13.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-22T20:06:13.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:06:13.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:06:13.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:06:13.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:06:13.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-22T20:06:13.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:06:13.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:06:13.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:06:13.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:13.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-22T20:06:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:13.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:13.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:3 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:3 2026-02-22T20:06:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-22T20:06:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:06:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:06:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:06:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:06:13.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:3 cluster2 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:3 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:3 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:13.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:13.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:06:13.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-22T20:06:13.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:06:13.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:06:13.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:06:13.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:06:13.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-22T20:06:13.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:06:13.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:06:13.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:06:13.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:13.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-22T20:06:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:13.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:13.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:13.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T20:06:14.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:06:14.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:06:14.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:14.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:14.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:06:14.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:14.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:14.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:06:14.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:06:14.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:14.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:06:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:06:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:06:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:06:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:06:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:06:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:06:14.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:06:14.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:06:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:06:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:06:14.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-22T20:06:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:14.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:14.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:14.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:14.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:06:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:06:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:06:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:06:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:06:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:06:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:06:14.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:14.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:06:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:06:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:06:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:06:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:06:14.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:06:14.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:06:14.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:06:14.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:06:14.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:14.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:06:14.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:06:14.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:06:14.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:06:14.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:06:14.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:14.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_force_promote 2026-02-22T20:06:14.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:14.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:14.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:14.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote started 2026-02-22T20:06:14.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:14.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:14.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:14.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:06:14.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:14.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:14.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:15.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_force_promote started 2026-02-22T20:06:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:06:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:06:15.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:06:15.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:06:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:06:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:06:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:06:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:06:15.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:15.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:06:15.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:15.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:15.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:06:15.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:15.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:15.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:15.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:06:15.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:15.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:15.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:06:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:06:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:06:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:15.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:06:15.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:15.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:15.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:15.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:15.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:15.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:15.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:15.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:06:15.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:06:15.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:15.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-22T20:06:15.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:15.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:06:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:06:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:15.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:06:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:06:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:06:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:15.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:06:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:06:15.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:06:15.739 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:15.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:15.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:15.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:15.747 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:15.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:06:15.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:06:15.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:06:15.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:06:15.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:06:15.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:06:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:06:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_force_promote 2026-02-22T20:06:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:06:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:06:15.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:06:15.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test_force_promote 2026-02-22T20:06:15.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:15.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:15.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:15.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test_force_promote 2026-02-22T20:06:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 148 2026-02-22T20:06:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test_force_promote 2026-02-22T20:06:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:06:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:16.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_force_promote.status 2026-02-22T20:06:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_force_promote.status 2026-02-22T20:06:16.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:16.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_force_promote.status 2026-02-22T20:06:16.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_force_promote.status 2026-02-22T20:06:16.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_force_promote.status 2026-02-22T20:06:16.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote.status 2026-02-22T20:06:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test_force_promote /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_force_promote.status 2026-02-22T20:06:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_force_promote.status 2026-02-22T20:06:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test_force_promote --format xml 2026-02-22T20:06:16.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:16.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:06:16.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=148 2026-02-22T20:06:16.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:06:16.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:16.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:06:16.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_force_promote /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote.status 2026-02-22T20:06:16.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:16.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:16.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:16.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote.status 2026-02-22T20:06:16.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_force_promote --format xml 2026-02-22T20:06:16.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:16.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:06:16.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=147 2026-02-22T20:06:16.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 148 = 147 2026-02-22T20:06:16.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:16.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:06:16.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_force_promote /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote.status 2026-02-22T20:06:16.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:16.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:16.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:16.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote.status 2026-02-22T20:06:16.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_force_promote --format xml 2026-02-22T20:06:16.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:16.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:06:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=148 2026-02-22T20:06:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 148 = 148 2026-02-22T20:06:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_force_promote 2026-02-22T20:06:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:16.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:06:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+replaying 2026-02-22T20:06:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:16.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:17.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+replaying '' '' 2026-02-22T20:06:17.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:17.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:17.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:17.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:17.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:17.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:17.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:17.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:17.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote 2026-02-22T20:06:17.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_force_promote: 2026-02-22T20:06:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 789b3e4e-b7a0-4613-8861-7cdd7d2076e6 2026-02-22T20:06:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:06:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790772,"replay_state":"idle"} 2026-02-22T20:06:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:06:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:14 2026-02-22T20:06:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:06:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:06:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:06:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:06:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:06:17.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:06:17.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:06:17.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:06:17.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790772,"replay_state":"idle"} 2026-02-22T20:06:17.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:17.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:06:17.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:06:17.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:06:17.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:06:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:06:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:06:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:06:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:14' 2026-02-22T20:06:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+replaying '2026-02-22 20:06:14' 2026-02-22T20:06:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:06:14' 2026-02-22T20:06:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:17.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:06:17.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING1143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790763,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:06:03cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:06:12test_force_promote789b3e4e-b7a0-4613-8861-7cdd7d2076e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790772,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 20:06:14cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:06:17.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:06:17.967 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2026-02-22T20:06:17.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:14' 2026-02-22T20:06:17.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2026-02-22T20:06:17.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:06:17.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:06:17.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:06:17.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:06:17.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:17.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:17.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2026-02-22T20:06:17.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:17.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:17.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:17.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:17.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:17.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:17.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:17.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:18.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped '' '' 2026-02-22T20:06:18.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:18.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:18.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:18.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:18.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:18.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:18.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:18.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:18.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote 2026-02-22T20:06:19.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_force_promote: 2026-02-22T20:06:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 789b3e4e-b7a0-4613-8861-7cdd7d2076e6 2026-02-22T20:06:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:06:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:06:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:06:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:17 2026-02-22T20:06:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:06:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:06:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:06:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790772,"replay_state":"idle"} 2026-02-22T20:06:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:14 2026-02-22T20:06:19.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:06:19.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 147 .mirror.primary.789b3e4e-b7a0-4613-8861-7cdd7d2076e6.dec4b0b8-5da7-42b1-8649-42755fffa031 (peer_uuids:[]) 2026-02-22T20:06:19.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 148 .mirror.primary.789b3e4e-b7a0-4613-8861-7cdd7d2076e6.28754ab2-6210-4bd0-a1e5-01277fa59ab7 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:06:19.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:06:19.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:06:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:06:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:06:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:06:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:06:19.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:06:19.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:06:19.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:06:19.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:06:19.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:06:19.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:17' 2026-02-22T20:06:19.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped '2026-02-22 20:06:17' 2026-02-22T20:06:19.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:19.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:19.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:19.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:19.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:06:17' 2026-02-22T20:06:19.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:19.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:19.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:06:19.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK244524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:06:17cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790763,"replay_state":"idle"}2026-02-22 20:06:03test_force_promote789b3e4e-b7a0-4613-8861-7cdd7d2076e6up+stoppedlocal image is primary44564466mirror.1trial0172026-02-22 20:06:17cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790772,"replay_state":"idle"}2026-02-22 20:06:14' 2026-02-22T20:06:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:06:19.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2026-02-22T20:06:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:17' 2026-02-22T20:06:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2026-02-22T20:06:19.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:06:19.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:06:19.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:06:19.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:06:19.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:19.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:19.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test_force_promote --force 2026-02-22T20:06:19.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:19.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:19.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:19.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force=--force 2026-02-22T20:06:19.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_force_promote --force 2026-02-22T20:06:23.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:06:23.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test_force_promote 2026-02-22T20:06:23.885 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:23.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:23.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:23.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote stopped 2026-02-22T20:06:23.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:23.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:23.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:23.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:06:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:24.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_force_promote stopped 2026-02-22T20:06:24.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:24.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:24.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:24.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:06:24.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:06:24.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:06:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:06:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:06:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:06:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:24.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:06:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:24.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:24.904 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:24.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-22T20:06:24.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:24.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:24.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:24.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:24.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:06:24.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:24.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:24.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:24.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:24.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:06:24.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:06:24.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:24.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:25.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:06:25.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:06:25.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:06:25.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:06:25.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:06:25.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:25.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote 2026-02-22T20:06:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped 2026-02-22T20:06:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:06:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:26.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped 2026-02-22T20:06:26.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:26.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:26.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:26.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:06:26.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:06:26.040 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:06:26.041 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:06:26.041 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:26.041 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:06:26.042 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:26.042 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:06:26.042 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:06:26.042 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:06:26.042 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:26.042 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:06:26.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:06:26.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:06:26.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:26.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:26.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:06:26.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:26.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:26.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:06:26.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:06:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:06:26.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:06:26.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:06:26.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:06:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:06:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:06:26.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:26.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2026-02-22T20:06:26.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:26.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2026-02-22T20:06:26.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:06:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:06:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:06:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:06:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:06:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:06:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:06:26.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:06:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:06:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:06:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:06:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:06:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:06:26.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:06:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:06:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:06:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:06:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:06:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:06:26.112 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:06:26.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:06:26.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:06:26.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:06:26.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2026-02-22T20:06:26.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:06:26.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:06:26.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:06:26.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:06:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:06:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:26.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+stopped 2026-02-22T20:06:26.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:26.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:26.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:26.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:26.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:26.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:26.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:26.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:27.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+stopped '' '' 2026-02-22T20:06:27.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:27.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:27.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:27.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:27.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:27.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:27.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:27.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:27.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote 2026-02-22T20:06:27.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_force_promote: 2026-02-22T20:06:27.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 789b3e4e-b7a0-4613-8861-7cdd7d2076e6 2026-02-22T20:06:27.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:06:27.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:06:27.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:06:27.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:23 2026-02-22T20:06:27.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:06:27.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:06:27.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:06:27.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:06:27.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:22 2026-02-22T20:06:27.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:06:27.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 150 .mirror.primary.789b3e4e-b7a0-4613-8861-7cdd7d2076e6.1aabf785-5784-42f0-83a4-898b92aece0a (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:06:27.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:06:27.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:06:27.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:06:27.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:06:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:06:27.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:06:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:06:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:06:27.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:06:27.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:06:27.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:06:27.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:23' 2026-02-22T20:06:27.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+stopped '2026-02-22 20:06:23' 2026-02-22T20:06:27.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:27.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:27.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:27.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:27.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:06:23' 2026-02-22T20:06:27.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:27.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:27.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK1143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:06:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:06:22test_force_promote789b3e4e-b7a0-4613-8861-7cdd7d2076e6up+stoppedlocal image is primary43424353mirror.1trial0172026-02-22 20:06:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:06:22' 2026-02-22T20:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:06:27.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2026-02-22T20:06:27.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:23' 2026-02-22T20:06:27.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2026-02-22T20:06:27.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:06:27.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:06:27.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:06:27.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:06:27.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:27.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:27.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2026-02-22T20:06:27.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:27.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:27.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:27.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:27.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:27.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:27.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:27.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:28.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped '' '' 2026-02-22T20:06:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:28.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote 2026-02-22T20:06:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_force_promote: 2026-02-22T20:06:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 789b3e4e-b7a0-4613-8861-7cdd7d2076e6 2026-02-22T20:06:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:06:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:06:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:06:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:27 2026-02-22T20:06:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:06:28.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:06:28.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:06:28.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:06:28.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:28 2026-02-22T20:06:28.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:06:28.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 147 .mirror.primary.789b3e4e-b7a0-4613-8861-7cdd7d2076e6.dec4b0b8-5da7-42b1-8649-42755fffa031 (peer_uuids:[]) 2026-02-22T20:06:28.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 148 .mirror.primary.789b3e4e-b7a0-4613-8861-7cdd7d2076e6.28754ab2-6210-4bd0-a1e5-01277fa59ab7 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:06:28.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:06:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:06:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:06:28.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:06:28.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:28.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:06:28.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:06:28.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:06:28.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:06:28.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:06:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:06:28.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:06:28.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:27' 2026-02-22T20:06:28.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped '2026-02-22 20:06:27' 2026-02-22T20:06:28.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:28.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:28.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:28.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:06:28.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:06:27' 2026-02-22T20:06:28.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:28.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:28.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:06:28.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK1144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:06:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}2026-02-22 20:06:23test_force_promote789b3e4e-b7a0-4613-8861-7cdd7d2076e6up+stoppedlocal image is primary44564466mirror.1trial0172026-02-22 20:06:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:06:28' 2026-02-22T20:06:28.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:06:28.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2026-02-22T20:06:28.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:27' 2026-02-22T20:06:28.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2026-02-22T20:06:28.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:06:28.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:06:28.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:06:28.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:06:28.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:28.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:28.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster1 mirror test_force_promote 100 2026-02-22T20:06:28.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:28.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:28.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:28.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:06:28.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:06:28.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:06:28.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:06:28.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:06:28.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:06:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:06:28.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 5000.21 bytes/sec: 20 MiB/s 2026-02-22T20:06:28.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_force_promote 100 2026-02-22T20:06:28.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:28.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:28.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:28.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:06:28.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:06:28.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:06:28.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:06:28.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:06:28.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:06:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:06:28.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 5000.21 bytes/sec: 20 MiB/s 2026-02-22T20:06:28.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster1 mirror test_force_promote 2026-02-22T20:06:28.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:28.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:28.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:28.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:06:28.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:06:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster1 mirror test_force_promote 2026-02-22T20:06:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:28.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_force_promote 2026-02-22T20:06:28.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 rm mirror/test_force_promote 2026-02-22T20:06:28.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:06:28.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:28.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test_force_promote 2026-02-22T20:06:28.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:06:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:06:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test_force_promote 2026-02-22T20:06:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote 2026-02-22T20:06:28.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_force_promote 2026-02-22T20:06:28.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test_force_promote 2026-02-22T20:06:30.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:06:30.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:30.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: cloned images' 2026-02-22T20:06:30.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:06:30.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:06:30.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:06:30 TEST: cloned images 2026-02-22T20:06:30.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:06:30 TEST: cloned images 2026-02-22T20:06:30.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog ' - default' 2026-02-22T20:06:30.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:06:30.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:06:30.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:06:30 - default 2026-02-22T20:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:06:30 - default 2026-02-22T20:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ parent_image=test_parent 2026-02-22T20:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ parent_snap=snap 2026-02-22T20:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror_parent test_parent snapshot 2026-02-22T20:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror_parent test_parent 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:06:30.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:30.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror_parent/test_parent 2026-02-22T20:06:30.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror_parent 2026-02-22T20:06:30.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:30.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:06:30.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:30.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror_parent test_parent snapshot 2026-02-22T20:06:30.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:30.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:30.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:30.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:06:30.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror_parent/test_parent snapshot 2026-02-22T20:06:31.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:06:31.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror_parent/test_parent 2026-02-22T20:06:31.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_parent': 2026-02-22T20:06:31.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:06:31.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:06:31.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:06:31.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 152715ded3ba 2026-02-22T20:06:31.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.152715ded3ba 2026-02-22T20:06:31.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:06:31.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:06:31.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:06:31.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:06:31.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:06:30 2026 2026-02-22T20:06:31.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:06:30 2026 2026-02-22T20:06:31.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:06:30 2026 2026-02-22T20:06:31.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 152715ded3ba 2026-02-22T20:06:31.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:06:31.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:06:31.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 565ee859-7c54-41ab-be1b-06a6f18f0a8e 2026-02-22T20:06:31.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:06:31.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror_parent test_parent 100 2026-02-22T20:06:31.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:31.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:31.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:31.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:06:31.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:06:31.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:06:31.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:06:31.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror_parent/test_parent --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:06:31.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:06:31.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:06:31.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 8333.68 bytes/sec: 33 MiB/s 2026-02-22T20:06:31.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap 2026-02-22T20:06:31.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:31.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:31.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:31.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap 2026-02-22T20:06:31.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap 2026-02-22T20:06:32.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:06:32.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ protect_snapshot cluster2 mirror_parent test_parent snap 2026-02-22T20:06:32.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:32.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:32.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:32.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap 2026-02-22T20:06:32.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap protect mirror_parent/test_parent@snap 2026-02-22T20:06:32.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ clone_image=test_clone 2026-02-22T20:06:32.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone 2026-02-22T20:06:32.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:32.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_pool=mirror_parent 2026-02-22T20:06:32.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_image=test_parent 2026-02-22T20:06:32.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_snap=snap 2026-02-22T20:06:32.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_pool=mirror 2026-02-22T20:06:32.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_image=test_clone 2026-02-22T20:06:32.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 6 2026-02-22T20:06:32.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone --image-feature layering,exclusive-lock,journaling 2026-02-22T20:06:32.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_clone 100 2026-02-22T20:06:32.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:32.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:32.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:06:32.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_clone --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:06:32.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:06:32.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:06:32.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 5000.21 bytes/sec: 20 MiB/s 2026-02-22T20:06:32.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:06:32.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test_clone snapshot 2026-02-22T20:06:32.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:32.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:32.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:32.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:06:32.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test_clone snapshot 2026-02-22T20:06:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:06:33.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test_clone 2026-02-22T20:06:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_clone': 2026-02-22T20:06:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:06:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:06:33.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:06:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 1539368c4057 2026-02-22T20:06:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.1539368c4057 2026-02-22T20:06:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:06:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:06:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:06:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:06:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:06:32 2026 2026-02-22T20:06:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:06:32 2026 2026-02-22T20:06:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:06:32 2026 2026-02-22T20:06:33.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: parent: mirror_parent/test_parent@snap 2026-02-22T20:06:33.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: overlap: 128 MiB 2026-02-22T20:06:33.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 1539368c4057 2026-02-22T20:06:33.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:06:33.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:06:33.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: b6a60180-5976-4d26-89cd-15efbf5fc1be 2026-02-22T20:06:33.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:06:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror_parent test_parent 2026-02-22T20:06:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror_parent test_parent started 2026-02-22T20:06:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:33.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:33.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:33.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:06:33.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:33.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:33.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:34.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror_parent test_parent started 2026-02-22T20:06:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:06:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:06:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:06:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror_parent/test_parent 2026-02-22T20:06:34.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:06:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:06:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:06:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:06:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:06:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:06:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2026-02-22T20:06:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:06:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:06:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:34.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:34.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:34.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:06:34.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:34.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:34.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:34.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:06:34.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:34.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:34.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:06:34.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:34.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:06:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:06:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:06:34.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:34.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:34.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:34.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2026-02-22T20:06:34.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2026-02-22T20:06:34.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:34.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:34.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:34.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2026-02-22T20:06:34.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:06:34.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:34.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:34.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:34.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:06:34.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:34.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:34.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:34.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:06:34.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:34.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:34.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:34.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:06:34.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:34.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:34.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:06:34.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:34.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:34.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:34.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:06:34.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:34.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:34.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:34.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:34.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:34.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:34.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:06:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:06:34.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:34.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:34.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:34.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:34.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2026-02-22T20:06:34.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:06:34.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:06:34.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:06:34.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:06:34.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:06:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:06:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:06:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror_parent test_parent 2026-02-22T20:06:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:06:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:34.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:34.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:06:34.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:06:34.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror_parent test_parent 2026-02-22T20:06:34.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:34.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:34.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:34.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror_parent/test_parent 2026-02-22T20:06:35.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 5 2026-02-22T20:06:35.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror_parent test_parent 2026-02-22T20:06:35.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:06:35.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:35.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:35.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:35.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror_parent-test_parent.status 2026-02-22T20:06:35.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror_parent-test_parent.status 2026-02-22T20:06:35.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror_parent-test_parent.status 2026-02-22T20:06:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror_parent-test_parent.status 2026-02-22T20:06:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror_parent-test_parent.status 2026-02-22T20:06:35.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:35.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.status 2026-02-22T20:06:35.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror_parent test_parent /tmp/tmp.bhaDYZBKee/cluster2-mirror_parent-test_parent.status 2026-02-22T20:06:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror_parent-test_parent.status 2026-02-22T20:06:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror_parent/test_parent --format xml 2026-02-22T20:06:35.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:35.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:06:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=5 2026-02-22T20:06:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:06:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:35.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:06:35.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror_parent test_parent /tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.status 2026-02-22T20:06:35.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:35.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:35.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:35.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.status 2026-02-22T20:06:35.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror_parent/test_parent --format xml 2026-02-22T20:06:35.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:35.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:06:35.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=3 2026-02-22T20:06:35.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 5 = 3 2026-02-22T20:06:35.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:35.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:06:35.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror_parent test_parent /tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.status 2026-02-22T20:06:35.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:35.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:35.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:35.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.status 2026-02-22T20:06:35.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror_parent/test_parent --format xml 2026-02-22T20:06:35.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:35.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:06:35.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:06:35.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:06:35.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:06:35.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:06:35.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:06:35.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 5 = '' 2026-02-22T20:06:35.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:35.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.8 2026-02-22T20:06:36.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror_parent test_parent /tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.status 2026-02-22T20:06:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.status 2026-02-22T20:06:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror_parent/test_parent --format xml 2026-02-22T20:06:36.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:36.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:06:36.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=5 2026-02-22T20:06:36.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 5 = 5 2026-02-22T20:06:36.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:36.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror_parent test_parent 2026-02-22T20:06:36.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:36.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:36.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:36.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:06:36.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying 2026-02-22T20:06:36.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:37.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying '' '' 2026-02-22T20:06:37.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:37.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:37.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:37.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:37.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:37.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:37.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:37.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:37.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror_parent/test_parent 2026-02-22T20:06:37.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_parent: 2026-02-22T20:06:37.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 565ee859-7c54-41ab-be1b-06a6f18f0a8e 2026-02-22T20:06:37.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:06:37.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790791,"replay_state":"idle"} 2026-02-22T20:06:37.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:06:37.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:33 2026-02-22T20:06:37.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:06:37.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:06:37.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:06:37.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:06:37.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:06:37.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:06:37.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:06:37.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:06:37.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790791,"replay_state":"idle"} 2026-02-22T20:06:37.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:37.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:06:37.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:06:37.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:06:37.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:06:37.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:06:37.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:06:37.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:06:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:33' 2026-02-22T20:06:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror_parent test_parent up+replaying '2026-02-22 20:06:33' 2026-02-22T20:06:37.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:06:33' 2026-02-22T20:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror_parent --verbose --format xml 2026-02-22T20:06:37.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING143384372mirror.0trial01719.2.3-689-gf90f4eaefalseOK43424365mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484373mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554371mirror.3trial01719.2.3-689-gf90f4eaetrueOKtest_parent565ee859-7c54-41ab-be1b-06a6f18f0a8eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790791,"replay_state":"idle"}43424365mirror.1trial0172026-02-22 20:06:33cluster2-DNE41c38dbe-ca8d-4a4d-903d-354708737fc5down+unknownstatus not found' 2026-02-22T20:06:37.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:06:37.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_parent'\'']/last_update' 2026-02-22T20:06:37.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:33' 2026-02-22T20:06:37.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_parent'\'']/state' 2026-02-22T20:06:37.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:06:37.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:06:37.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:06:37.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:06:37.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:37.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:37.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror_parent test_parent 2026-02-22T20:06:37.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:37.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:37.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:06:37.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror_parent-test_parent.export 2026-02-22T20:06:37.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror_parent-test_parent.export 2026-02-22T20:06:37.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:37.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror_parent-test_parent.export 2026-02-22T20:06:37.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror_parent-test_parent.export 2026-02-22T20:06:37.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror_parent-test_parent.export 2026-02-22T20:06:37.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:37.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.export 2026-02-22T20:06:37.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror_parent-test_parent.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.export 2026-02-22T20:06:37.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror_parent/test_parent /tmp/tmp.bhaDYZBKee/cluster2-mirror_parent-test_parent.export 2026-02-22T20:06:37.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:37.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror_parent/test_parent /tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.export 2026-02-22T20:06:38.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:38.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror_parent-test_parent.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.export 2026-02-22T20:06:38.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror_parent-test_parent.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_parent-test_parent.export 2026-02-22T20:06:38.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:38.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone 2026-02-22T20:06:38.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:38.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:38.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:38.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone started 2026-02-22T20:06:38.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:38.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:38.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:38.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:06:38.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:38.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:38.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_clone started 2026-02-22T20:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:06:39.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone 2026-02-22T20:06:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:06:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:06:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:06:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:06:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:06:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:39.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:06:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone 2026-02-22T20:06:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:06:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:06:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:39.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:39.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:06:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:06:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:06:39.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:39.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:39.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:39.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:39.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone 2026-02-22T20:06:39.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2026-02-22T20:06:39.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:39.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone 2026-02-22T20:06:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:06:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:06:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:06:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:06:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:39.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:39.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:06:39.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:06:39.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:39.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:39.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:39.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:39.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone 2026-02-22T20:06:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:06:39.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:06:39.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:06:39.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:06:39.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:06:39.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:06:39.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:06:39.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:39.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_clone 2026-02-22T20:06:39.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:06:39.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:39.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:39.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:39.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:06:39.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:06:39.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test_clone 2026-02-22T20:06:39.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:39.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:39.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:39.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test_clone 2026-02-22T20:06:40.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 150 2026-02-22T20:06:40.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test_clone 2026-02-22T20:06:40.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:06:40.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:40.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:40.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:40.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_clone.status 2026-02-22T20:06:40.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_clone.status 2026-02-22T20:06:40.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_clone.status 2026-02-22T20:06:40.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_clone.status 2026-02-22T20:06:40.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_clone.status 2026-02-22T20:06:40.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:40.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_clone.status 2026-02-22T20:06:40.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test_clone /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_clone.status 2026-02-22T20:06:40.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:40.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:40.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:40.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_clone.status 2026-02-22T20:06:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test_clone --format xml 2026-02-22T20:06:40.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:40.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:06:40.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=150 2026-02-22T20:06:40.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:06:40.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:40.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:06:40.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_clone /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_clone.status 2026-02-22T20:06:40.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:40.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:40.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:40.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_clone.status 2026-02-22T20:06:40.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_clone --format xml 2026-02-22T20:06:40.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:40.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:06:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=149 2026-02-22T20:06:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 150 = 149 2026-02-22T20:06:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:06:40.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:06:40.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_clone /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_clone.status 2026-02-22T20:06:40.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:40.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:40.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:40.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_clone.status 2026-02-22T20:06:40.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_clone --format xml 2026-02-22T20:06:40.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:06:40.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:06:40.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=150 2026-02-22T20:06:40.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 150 = 150 2026-02-22T20:06:40.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_clone 2026-02-22T20:06:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:06:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_clone up+replaying 2026-02-22T20:06:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:40.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:40.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:40.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:40.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:41.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_clone up+replaying '' '' 2026-02-22T20:06:41.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:41.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:41.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:41.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:41.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:06:41.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:06:41.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:41.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:41.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_clone 2026-02-22T20:06:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_clone: 2026-02-22T20:06:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: b6a60180-5976-4d26-89cd-15efbf5fc1be 2026-02-22T20:06:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:06:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790793,"replay_state":"idle"} 2026-02-22T20:06:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:06:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:06:38 2026-02-22T20:06:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:06:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:06:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:06:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:06:41.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:06:41.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:06:41.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:06:41.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:06:41.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790793,"replay_state":"idle"} 2026-02-22T20:06:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:06:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:06:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:06:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:06:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:06:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:06:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:06:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:06:41.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:38' 2026-02-22T20:06:41.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_clone up+replaying '2026-02-22 20:06:38' 2026-02-22T20:06:41.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:41.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:41.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:41.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:06:41.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:06:38' 2026-02-22T20:06:41.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:06:41.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:06:41.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:06:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING1143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:06:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:06:37test_cloneb6a60180-5976-4d26-89cd-15efbf5fc1beup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790793,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 20:06:38cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:06:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:06:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_clone'\'']/last_update' 2026-02-22T20:06:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:06:38' 2026-02-22T20:06:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_clone'\'']/state' 2026-02-22T20:06:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:06:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:06:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:06:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:06:41.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:41.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:41.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test_clone 2026-02-22T20:06:41.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:41.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:41.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:06:41.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_clone.export 2026-02-22T20:06:41.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_clone.export 2026-02-22T20:06:41.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:41.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_clone.export 2026-02-22T20:06:41.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_clone.export 2026-02-22T20:06:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_clone.export 2026-02-22T20:06:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:06:41.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_clone.export 2026-02-22T20:06:41.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_clone.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_clone.export 2026-02-22T20:06:41.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test_clone /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_clone.export 2026-02-22T20:06:42.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:42.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test_clone /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_clone.export 2026-02-22T20:06:42.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:06:42.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_clone.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_clone.export 2026-02-22T20:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_clone.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_clone.export 2026-02-22T20:06:42.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:42.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test_clone 2026-02-22T20:06:42.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:42.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:42.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:42.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:06:42.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:06:42.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test_clone 2026-02-22T20:06:42.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:42.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:42.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone 2026-02-22T20:06:42.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone 2026-02-22T20:06:42.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone 2026-02-22T20:06:44.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:06:44.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:44.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog ' - clone v1' 2026-02-22T20:06:44.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:06:44.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:06:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:06:44 - clone v1 2026-02-22T20:06:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:06:44 - clone v1 2026-02-22T20:06:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ clone_image_and_enable_mirror cluster1 mirror_parent test_parent snap mirror test_clone1 snapshot 2026-02-22T20:06:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_pool=mirror_parent 2026-02-22T20:06:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_image=test_parent 2026-02-22T20:06:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_snap=snap 2026-02-22T20:06:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_pool=mirror 2026-02-22T20:06:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_image=test_clone1 2026-02-22T20:06:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 6 2026-02-22T20:06:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:06:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:06:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ clone_image cluster1 mirror_parent test_parent snap mirror test_clone1 2026-02-22T20:06:44.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:44.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_pool=mirror_parent 2026-02-22T20:06:44.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_image=test_parent 2026-02-22T20:06:44.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_snap=snap 2026-02-22T20:06:44.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_pool=mirror 2026-02-22T20:06:44.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_image=test_clone1 2026-02-22T20:06:44.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 6 2026-02-22T20:06:44.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap mirror/test_clone1 --image-feature layering,exclusive-lock,journaling 2026-02-22T20:06:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:06:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:06:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster1 mirror test_clone1 snapshot 2026-02-22T20:06:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:44.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:44.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone1 2026-02-22T20:06:44.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:06:44.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/test_clone1 snapshot 2026-02-22T20:06:44.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:06:44.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 info mirror/test_clone1 2026-02-22T20:06:44.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_clone1': 2026-02-22T20:06:44.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:06:44.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:06:44.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 156557c23e9b 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.156557c23e9b 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:06:44 2026 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:06:44 2026 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:06:44 2026 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: parent: mirror_parent/test_parent@snap 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: overlap: 128 MiB 2026-02-22T20:06:44.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 156557c23e9b 2026-02-22T20:06:44.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:06:44.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:06:44.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: d96b585f-d268-4952-a0ab-df49869c715d 2026-02-22T20:06:44.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:06:44.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ clone_image_and_enable_mirror cluster2 mirror_parent test_parent snap mirror test_clone_v1 snapshot --rbd-default-clone-format 1 2026-02-22T20:06:44.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:44.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_pool=mirror_parent 2026-02-22T20:06:44.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_image=test_parent 2026-02-22T20:06:44.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_snap=snap 2026-02-22T20:06:44.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_pool=mirror 2026-02-22T20:06:44.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_image=test_clone_v1 2026-02-22T20:06:44.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 6 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone_v1 --rbd-default-clone-format 1 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_pool=mirror_parent 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_image=test_parent 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_snap=snap 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_pool=mirror 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_image=test_clone_v1 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 6 2026-02-22T20:06:44.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone_v1 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 1 2026-02-22T20:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test_clone_v1 snapshot 2026-02-22T20:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v1 2026-02-22T20:06:44.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:06:44.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test_clone_v1 snapshot 2026-02-22T20:06:45.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:06:45.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test_clone_v1 2026-02-22T20:06:45.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_clone_v1': 2026-02-22T20:06:45.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:06:45.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:06:45.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:06:45.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 155d892db1fb 2026-02-22T20:06:45.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.155d892db1fb 2026-02-22T20:06:45.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:06:45.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:06:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:06:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:06:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:06:44 2026 2026-02-22T20:06:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:06:44 2026 2026-02-22T20:06:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:06:44 2026 2026-02-22T20:06:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: parent: mirror_parent/test_parent@snap 2026-02-22T20:06:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: overlap: 128 MiB 2026-02-22T20:06:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 155d892db1fb 2026-02-22T20:06:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:06:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:06:45.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: de92e91c-5f78-4085-a9c9-a61b061a6901 2026-02-22T20:06:45.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:06:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_clone_format cluster2 mirror test_clone_v1 2026-02-22T20:06:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:06:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:06:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_clone_v1 2026-02-22T20:06:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 info mirror/test_clone_v1 2026-02-22T20:06:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ awk 'BEGIN { 2026-02-22T20:06:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: format = 1 2026-02-22T20:06:45.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: $1 == "parent:" { 2026-02-22T20:06:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: parent = $2 2026-02-22T20:06:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: /op_features: .*clone-child/ { 2026-02-22T20:06:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: format = 2 2026-02-22T20:06:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: END { 2026-02-22T20:06:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: if (!parent) exit 1 2026-02-22T20:06:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: print format 2026-02-22T20:06:45.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: }' 2026-02-22T20:06:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 1 = 1 2026-02-22T20:06:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v1 2026-02-22T20:06:45.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v1 2026-02-22T20:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v1 started 2026-02-22T20:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v1 2026-02-22T20:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_clone_v1 started 2026-02-22T20:06:46.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:46.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:46.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v1 2026-02-22T20:06:46.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:06:46.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:06:46.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:06:46.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone_v1 2026-02-22T20:06:46.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:06:46.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:06:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:06:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:06:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:46.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:06:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:46.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:06:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2026-02-22T20:06:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:06:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:46.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:06:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:06:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:06:46.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:46.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:06:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:06:46.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:46.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:46.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:46.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:46.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2026-02-22T20:06:46.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2026-02-22T20:06:46.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:46.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1 2026-02-22T20:06:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:06:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:46.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:46.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:06:46.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:06:46.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:46.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:46.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1 2026-02-22T20:06:46.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2026-02-22T20:06:46.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:46.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v1 2026-02-22T20:06:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:06:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:06:46.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:06:46.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:06:46.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:06:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:46.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v1 2026-02-22T20:06:46.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:06:46.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:06:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:06:46.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:06:46.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:06:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:06:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:06:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_clone_format cluster1 mirror test_clone_v1 2026-02-22T20:06:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:06:46.655 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_clone_v1 2026-02-22T20:06:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 info mirror/test_clone_v1 2026-02-22T20:06:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ awk 'BEGIN { 2026-02-22T20:06:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: format = 1 2026-02-22T20:06:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: $1 == "parent:" { 2026-02-22T20:06:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: parent = $2 2026-02-22T20:06:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: /op_features: .*clone-child/ { 2026-02-22T20:06:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: format = 2 2026-02-22T20:06:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: END { 2026-02-22T20:06:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: if (!parent) exit 1 2026-02-22T20:06:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: print format 2026-02-22T20:06:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: }' 2026-02-22T20:06:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 1 = 1 2026-02-22T20:06:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test_clone_v1 2026-02-22T20:06:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v1 2026-02-22T20:06:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:06:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:06:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test_clone_v1 2026-02-22T20:06:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v1 2026-02-22T20:06:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone_v1 2026-02-22T20:06:46.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone_v1 2026-02-22T20:06:47.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:06:47.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:47.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster1 mirror test_clone1 2026-02-22T20:06:47.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:47.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:47.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone1 2026-02-22T20:06:47.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:06:47.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:06:47.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster1 mirror test_clone1 2026-02-22T20:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone1 2026-02-22T20:06:47.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_clone1 2026-02-22T20:06:47.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 rm mirror/test_clone1 2026-02-22T20:06:47.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:06:47.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:47.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ unprotect_snapshot_retry cluster2 mirror_parent test_parent snap 2026-02-22T20:06:47.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:47.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:47.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:47.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap 2026-02-22T20:06:47.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:06:47.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:06:47.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ unprotect_snapshot cluster2 mirror_parent test_parent snap 2026-02-22T20:06:47.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:47.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:47.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:47.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap 2026-02-22T20:06:47.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap unprotect mirror_parent/test_parent@snap 2026-02-22T20:06:47.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: unprotecting snap failed: (16) Device or resource busy 2026-02-22T20:06:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:06:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ unprotect_snapshot cluster2 mirror_parent test_parent snap 2026-02-22T20:06:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap 2026-02-22T20:06:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap unprotect mirror_parent/test_parent@snap 2026-02-22T20:06:48.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:48.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap 2026-02-22T20:06:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap 2026-02-22T20:06:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap 2026-02-22T20:06:49.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:49.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog ' - clone v2' 2026-02-22T20:06:49.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:06:49.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:06:49.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:06:49 - clone v2 2026-02-22T20:06:49.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:06:49 - clone v2 2026-02-22T20:06:49.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ parent_snap=snap_v2 2026-02-22T20:06:49.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2 2026-02-22T20:06:49.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:49.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:49.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:49.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_v2 2026-02-22T20:06:49.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2 2026-02-22T20:06:50.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:06:50.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:06:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror_parent test_parent 2026-02-22T20:06:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:50.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror_parent/test_parent 2026-02-22T20:06:52.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 7 2026-02-22T20:06:52.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ clone_image_and_enable_mirror cluster2 mirror_parent test_parent snap_v2 mirror test_clone_v2 snapshot --rbd-default-clone-format 2 2026-02-22T20:06:52.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:52.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_pool=mirror_parent 2026-02-22T20:06:52.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_image=test_parent 2026-02-22T20:06:52.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_snap=snap_v2 2026-02-22T20:06:52.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_pool=mirror 2026-02-22T20:06:52.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_image=test_clone_v2 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 6 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ clone_image cluster2 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_pool=mirror_parent 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_image=test_parent 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_snap=snap_v2 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_pool=mirror 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_image=test_clone_v2 2026-02-22T20:06:52.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 6 2026-02-22T20:06:52.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2 2026-02-22T20:06:52.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:06:52.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:52.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:06:52.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:52.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test_clone_v2 snapshot 2026-02-22T20:06:52.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:52.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:52.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v2 2026-02-22T20:06:52.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:06:52.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test_clone_v2 snapshot 2026-02-22T20:06:53.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:06:53.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test_clone_v2 2026-02-22T20:06:53.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_clone_v2': 2026-02-22T20:06:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:06:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:06:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:06:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 157eebf6ee88 2026-02-22T20:06:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.157eebf6ee88 2026-02-22T20:06:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:06:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling, operations 2026-02-22T20:06:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: clone-child 2026-02-22T20:06:53.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:06:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:06:52 2026 2026-02-22T20:06:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:06:52 2026 2026-02-22T20:06:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:06:52 2026 2026-02-22T20:06:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: parent: mirror_parent/test_parent@snap_v2 2026-02-22T20:06:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: overlap: 128 MiB 2026-02-22T20:06:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 157eebf6ee88 2026-02-22T20:06:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:06:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:06:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: ee759409-76c2-4eb9-a6e1-819cb5bcaec9 2026-02-22T20:06:53.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:06:53.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_clone_format cluster2 mirror test_clone_v2 2026-02-22T20:06:53.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:06:53.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:06:53.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_clone_v2 2026-02-22T20:06:53.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 info mirror/test_clone_v2 2026-02-22T20:06:53.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ awk 'BEGIN { 2026-02-22T20:06:53.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: format = 1 2026-02-22T20:06:53.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:53.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: $1 == "parent:" { 2026-02-22T20:06:53.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: parent = $2 2026-02-22T20:06:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: /op_features: .*clone-child/ { 2026-02-22T20:06:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: format = 2 2026-02-22T20:06:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: END { 2026-02-22T20:06:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: if (!parent) exit 1 2026-02-22T20:06:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: print format 2026-02-22T20:06:53.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: }' 2026-02-22T20:06:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 2 = 2 2026-02-22T20:06:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v2 2026-02-22T20:06:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v2 2026-02-22T20:06:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v2 started 2026-02-22T20:06:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:53.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:53.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v2 2026-02-22T20:06:53.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:06:53.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:53.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:53.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:54.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_clone_v2 started 2026-02-22T20:06:54.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:54.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:54.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v2 2026-02-22T20:06:54.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:06:54.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:06:54.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:06:54.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:54.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:06:54.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:06:54.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:54.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:54.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:06:54.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:06:54.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:54.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:06:54.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:54.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:06:54.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:54.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:54.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:54.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:54.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:54.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:54.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:54.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:06:54.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:06:54.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:54.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:54.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:54.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:54.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-22T20:06:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:54.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:54.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:54.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:54.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:06:54.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:54.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:54.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:54.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:06:54.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:06:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:06:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:06:54.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:54.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:54.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:54.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:54.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:54.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-22T20:06:54.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:54.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:06:54.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:06:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:06:54.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:06:54.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:06:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:54.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-22T20:06:54.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:54.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:54.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:54.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:54.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:54.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:06:54.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:06:54.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:54.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:54.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:06:54.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:06:54.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:06:54.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:54.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:54.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:06:54.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:54.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-22T20:06:54.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:54.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:54.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:06:54.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T20:06:55.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:06:55.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:06:55.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:55.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:55.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:06:55.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:55.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:55.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:06:55.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:06:55.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:55.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:06:55.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:06:55.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:55.537 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:55.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:06:55.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:55.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2026-02-22T20:06:55.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:06:55.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:06:55.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:06:55.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:55.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:06:55.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:06:55.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:06:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:06:55.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:06:55.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:06:55.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:06:55.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:06:55.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:06:55.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:06:55.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:06:55.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:55.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:55.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:06:55.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2026-02-22T20:06:55.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:06:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:06:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:06:55.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:06:55.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:06:55.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:06:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:06:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_clone_format cluster1 mirror test_clone_v2 2026-02-22T20:06:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:06:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_clone_v2 2026-02-22T20:06:55.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 info mirror/test_clone_v2 2026-02-22T20:06:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ awk 'BEGIN { 2026-02-22T20:06:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: format = 1 2026-02-22T20:06:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: $1 == "parent:" { 2026-02-22T20:06:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: parent = $2 2026-02-22T20:06:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: /op_features: .*clone-child/ { 2026-02-22T20:06:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: format = 2 2026-02-22T20:06:55.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: } 2026-02-22T20:06:55.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: END { 2026-02-22T20:06:55.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: if (!parent) exit 1 2026-02-22T20:06:55.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: print format 2026-02-22T20:06:55.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: }' 2026-02-22T20:06:55.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 2 = 2 2026-02-22T20:06:55.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2 2026-02-22T20:06:55.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:55.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:55.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:55.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_v2 2026-02-22T20:06:55.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2 2026-02-22T20:06:55.752 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing snap: 100% complete...done. 2026-02-22T20:06:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:06:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror_parent test_parent 2026-02-22T20:06:55.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:55.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:55.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:55.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror_parent/test_parent 2026-02-22T20:06:57.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 8 2026-02-22T20:06:57.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2026-02-22T20:06:57.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:57.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:57.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:57.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap_v2 2026-02-22T20:06:57.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2026-02-22T20:06:57.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-22T20:06:57.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 6 f68bc9a9-5434-4a3b-a3ee-dd9ff19285e7 128 MiB Sun Feb 22 20:06:50 2026 trash (user snap_v2) 2026-02-22T20:06:57.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-22T20:06:57.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:57.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:06:57.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:06:58.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-22T20:06:58.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:58.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:58.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:58.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap_v2 2026-02-22T20:06:58.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2026-02-22T20:06:58.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-22T20:06:58.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 6 db2e0f11-95c8-4d43-bbe9-02aa105568a0 128 MiB Sun Feb 22 20:06:51 2026 trash (user snap_v2) 2026-02-22T20:06:58.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:58.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test_clone_v2 2026-02-22T20:06:58.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:58.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:58.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v2 2026-02-22T20:06:58.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:06:58.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:06:58.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test_clone_v2 2026-02-22T20:06:58.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:58.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:58.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v2 2026-02-22T20:06:58.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone_v2 2026-02-22T20:06:58.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone_v2 2026-02-22T20:06:59.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:06:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test_clone_v2 deleted 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v2 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test_clone_v2 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_clone_v2 2026-02-22T20:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test_clone_v2 2026-02-22T20:06:59.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:06:59.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory 2026-02-22T20:06:59.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id= 2026-02-22T20:06:59.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:06:59.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:06:59.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test_clone_v2 deleted '' 2026-02-22T20:06:59.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:06:59.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:06:59.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v2 2026-02-22T20:06:59.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:06:59.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:06:59.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:06:59.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:06:59.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test_clone_v2 2026-02-22T20:06:59.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:06:59.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:06:59.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_clone_v2 2026-02-22T20:06:59.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test_clone_v2 2026-02-22T20:06:59.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:06:59.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory 2026-02-22T20:06:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:06:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:06:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T20:06:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2 2026-02-22T20:06:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2026-02-22T20:06:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:06:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:06:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:06:59.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap_v2 2026-02-22T20:06:59.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2026-02-22T20:06:59.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-22T20:06:59.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:06:59.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2026-02-22T20:06:59.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:06:59.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:06:59.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:07:00.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2026-02-22T20:07:00.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-22T20:07:00.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:00.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:07:00.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:07:00.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap_v2 2026-02-22T20:07:00.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2026-02-22T20:07:00.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-22T20:07:00.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:00.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:00.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog ' - clone v2 non-primary' 2026-02-22T20:07:00.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:07:00.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:07:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:07:00 - clone v2 non-primary 2026-02-22T20:07:00.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:07:00 - clone v2 non-primary 2026-02-22T20:07:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2 2026-02-22T20:07:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:07:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:07:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_v2 2026-02-22T20:07:00.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2 2026-02-22T20:07:01.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:07:01.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:07:01.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror_parent test_parent 2026-02-22T20:07:01.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:01.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:07:01.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:07:01.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror_parent/test_parent 2026-02-22T20:07:03.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 10 2026-02-22T20:07:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snap_present cluster1 mirror_parent test_parent snap_v2 2026-02-22T20:07:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:07:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:07:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap_v2 2026-02-22T20:07:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:07:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:07:04.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror_parent/test_parent@snap_v2 2026-02-22T20:07:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_parent': 2026-02-22T20:07:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:07:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:07:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 3 2026-02-22T20:07:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 110d950ac276 2026-02-22T20:07:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.110d950ac276 2026-02-22T20:07:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:07:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling, operations, non-primary 2026-02-22T20:07:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: snap-trash 2026-02-22T20:07:04.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:07:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:06:33 2026 2026-02-22T20:07:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:06:33 2026 2026-02-22T20:07:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:06:33 2026 2026-02-22T20:07:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: protected: False 2026-02-22T20:07:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 110d950ac276 2026-02-22T20:07:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:07:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:07:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 565ee859-7c54-41ab-be1b-06a6f18f0a8e 2026-02-22T20:07:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: false 2026-02-22T20:07:04.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:04.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ clone_image_and_enable_mirror cluster1 mirror_parent test_parent snap_v2 mirror test_clone_v2 snapshot --rbd-default-clone-format 2 2026-02-22T20:07:04.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_pool=mirror_parent 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_image=test_parent 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_snap=snap_v2 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_pool=mirror 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_image=test_clone_v2 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 6 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ clone_image cluster1 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_pool=mirror_parent 2026-02-22T20:07:04.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_image=test_parent 2026-02-22T20:07:04.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local parent_snap=snap_v2 2026-02-22T20:07:04.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_pool=mirror 2026-02-22T20:07:04.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local clone_image=test_clone_v2 2026-02-22T20:07:04.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 6 2026-02-22T20:07:04.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2 2026-02-22T20:07:04.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:04.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:04.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:07:04.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:04.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster1 mirror test_clone_v2 snapshot 2026-02-22T20:07:04.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v2 2026-02-22T20:07:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/test_clone_v2 snapshot 2026-02-22T20:07:04.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:07:04.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 info mirror/test_clone_v2 2026-02-22T20:07:04.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_clone_v2': 2026-02-22T20:07:04.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 1586f6259afa 2026-02-22T20:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.1586f6259afa 2026-02-22T20:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling, operations 2026-02-22T20:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: clone-child 2026-02-22T20:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:07:04 2026 2026-02-22T20:07:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:07:04 2026 2026-02-22T20:07:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:07:04 2026 2026-02-22T20:07:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: parent: mirror_parent/test_parent@snap_v2 2026-02-22T20:07:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: overlap: 128 MiB 2026-02-22T20:07:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 1586f6259afa 2026-02-22T20:07:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:07:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:07:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 106e5008-455a-43d2-b55d-59fbf5b59c19 2026-02-22T20:07:04.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:07:04.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2 2026-02-22T20:07:04.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:04.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:07:04.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:07:04.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_v2 2026-02-22T20:07:04.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2 2026-02-22T20:07:05.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing snap: 100% complete...done. 2026-02-22T20:07:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2 2026-02-22T20:07:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2026-02-22T20:07:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:07:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:07:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap_v2 2026-02-22T20:07:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2026-02-22T20:07:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-22T20:07:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:07:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror_parent test_parent 2026-02-22T20:07:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:07:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:07:05.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror_parent/test_parent 2026-02-22T20:07:07.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 11 2026-02-22T20:07:07.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-22T20:07:07.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:07.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:07:07.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:07:08.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-22T20:07:08.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:08.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:07:08.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:07:08.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap_v2 2026-02-22T20:07:08.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2026-02-22T20:07:08.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-22T20:07:08.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 9 04c13a90-e8a5-473f-ba83-121982856b81 128 MiB Sun Feb 22 20:07:02 2026 trash (user snap_v2) 2026-02-22T20:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster1 mirror test_clone_v2 2026-02-22T20:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v2 2026-02-22T20:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:07:08.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:07:08.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster1 mirror test_clone_v2 2026-02-22T20:07:08.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:08.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:08.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_clone_v2 2026-02-22T20:07:08.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_clone_v2 2026-02-22T20:07:08.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 rm mirror/test_clone_v2 2026-02-22T20:07:09.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:07:09.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:09.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2026-02-22T20:07:09.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:09.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:07:09.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:07:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2026-02-22T20:07:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2026-02-22T20:07:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:07:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:07:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap_v2 2026-02-22T20:07:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2026-02-22T20:07:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -F ' trash (user snap_v2)' 2026-02-22T20:07:10.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:10.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:10.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror_parent test_parent 2026-02-22T20:07:10.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:10.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:07:10.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:07:10.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:07:10.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:07:10.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror_parent test_parent 2026-02-22T20:07:10.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:10.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror_parent 2026-02-22T20:07:10.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_parent 2026-02-22T20:07:10.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror_parent/test_parent 2026-02-22T20:07:10.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror_parent/test_parent 2026-02-22T20:07:12.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:07:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: data pool' 2026-02-22T20:07:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:07:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:07:12.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:07:12 TEST: data pool 2026-02-22T20:07:12.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:07:12 TEST: data pool 2026-02-22T20:07:12.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ dp_image=test_data_pool 2026-02-22T20:07:12.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test_data_pool snapshot 128 --data-pool mirror_parent 2026-02-22T20:07:12.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:07:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test_data_pool 128 --data-pool mirror_parent 2026-02-22T20:07:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:07:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 128 ']' 2026-02-22T20:07:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=128 2026-02-22T20:07:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:12.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/test_data_pool 2026-02-22T20:07:12.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:12.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:12.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:07:12.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:12.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test_data_pool snapshot 2026-02-22T20:07:12.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:12.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:12.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:12.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:12.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test_data_pool snapshot 2026-02-22T20:07:13.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:07:13.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test_data_pool 2026-02-22T20:07:13.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_data_pool': 2026-02-22T20:07:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:07:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:07:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:07:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 15b150d30764 2026-02-22T20:07:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: data_pool: mirror_parent 2026-02-22T20:07:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.3.15b150d30764 2026-02-22T20:07:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:07:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling, data-pool 2026-02-22T20:07:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:07:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:07:13.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:07:12 2026 2026-02-22T20:07:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:07:12 2026 2026-02-22T20:07:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:07:12 2026 2026-02-22T20:07:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 15b150d30764 2026-02-22T20:07:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:07:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:07:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: bf0a2bc0-a879-4cc2-80f7-6ae2bcf3c1fd 2026-02-22T20:07:13.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:07:13.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_data_pool cluster2 mirror test_data_pool 2026-02-22T20:07:13.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:07:13.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:07:13.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_data_pool 2026-02-22T20:07:13.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 info mirror/test_data_pool 2026-02-22T20:07:13.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ awk '$1 == "data_pool:" {print $2}' 2026-02-22T20:07:13.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ data_pool=mirror_parent 2026-02-22T20:07:13.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test mirror_parent = mirror_parent 2026-02-22T20:07:13.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_data_pool 2026-02-22T20:07:13.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:13.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:13.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:13.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_data_pool started 2026-02-22T20:07:13.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:13.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:13.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:07:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:07:13.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:07:14.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_data_pool started 2026-02-22T20:07:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:07:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:07:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:07:14.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_data_pool 2026-02-22T20:07:14.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:07:14.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:07:14.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:07:14.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:07:14.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:07:14.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:07:14.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:07:14.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:07:14.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:07:14.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:07:14.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:07:14.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2026-02-22T20:07:14.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:07:14.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:07:14.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:07:14.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:07:14.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:07:14.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:07:14.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:07:14.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:07:14.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:07:14.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:07:14.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2026-02-22T20:07:14.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-22T20:07:14.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:07:14.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:07:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:07:14.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:07:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:07:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:07:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:07:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2026-02-22T20:07:14.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-22T20:07:14.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:07:14.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_data_pool 2026-02-22T20:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:07:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:07:14.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:07:14.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:07:14.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:07:14.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_data_pool 2026-02-22T20:07:14.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-22T20:07:14.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:07:14.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:07:14.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:07:14.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_data_pool 2026-02-22T20:07:14.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:07:14.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:07:14.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:07:14.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:07:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:07:14.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:07:14.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:07:14.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:07:14.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:07:14.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:07:14.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:07:14.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_data_pool 2026-02-22T20:07:14.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-22T20:07:14.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:07:14.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:07:14.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:07:14.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T20:07:15.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:07:15.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:07:15.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:07:15.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2026-02-22T20:07:15.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:07:15.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:07:15.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:07:15.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:07:15.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:07:15.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:07:15.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:07:15.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:07:15.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:07:15.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:07:15.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:07:15.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:07:15.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:07:15.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2026-02-22T20:07:15.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2026-02-22T20:07:15.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:07:15.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:07:15.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:07:15.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2026-02-22T20:07:15.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:07:15.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:07:15.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:07:15.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:07:15.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:07:15.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:07:15.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:07:15.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:07:15.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:07:15.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2026-02-22T20:07:15.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:07:15.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:07:15.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:07:15.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:07:15.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:07:15.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:07:15.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:07:15.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:15.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_data_pool cluster1 mirror test_data_pool 2026-02-22T20:07:15.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:07:15.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:07:15.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_data_pool 2026-02-22T20:07:15.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 info mirror/test_data_pool 2026-02-22T20:07:15.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ awk '$1 == "data_pool:" {print $2}' 2026-02-22T20:07:15.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ data_pool=mirror_parent 2026-02-22T20:07:15.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test mirror_parent = mirror_parent 2026-02-22T20:07:15.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test_data_pool snap1 2026-02-22T20:07:15.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:15.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:15.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:15.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:07:15.663 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap1 2026-02-22T20:07:16.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:07:16.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_data_pool 100 2026-02-22T20:07:16.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:16.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:07:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:07:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:07:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:07:16.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:07:16.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:07:16.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.24 bytes/sec: 24 MiB/s 2026-02-22T20:07:16.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test_data_pool snap2 2026-02-22T20:07:16.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:16.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:16.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:16.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap2 2026-02-22T20:07:16.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap2 2026-02-22T20:07:17.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:07:17.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_data_pool 100 2026-02-22T20:07:17.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:17.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:17.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:17.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:07:17.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:07:17.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:17.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:07:17.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:07:17.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:07:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:07:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.24 bytes/sec: 24 MiB/s 2026-02-22T20:07:17.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_data_pool 2026-02-22T20:07:17.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:07:17.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:17.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:17.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:17.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:07:17.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:07:17.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test_data_pool 2026-02-22T20:07:17.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:17.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:17.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:17.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test_data_pool 2026-02-22T20:07:18.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 15 2026-02-22T20:07:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test_data_pool 2026-02-22T20:07:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:07:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:18.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_data_pool.status 2026-02-22T20:07:18.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_data_pool.status 2026-02-22T20:07:18.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:07:18.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool.status 2026-02-22T20:07:18.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_data_pool.status 2026-02-22T20:07:18.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_data_pool.status 2026-02-22T20:07:18.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:07:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.status 2026-02-22T20:07:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test_data_pool /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool.status 2026-02-22T20:07:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool.status 2026-02-22T20:07:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test_data_pool --format xml 2026-02-22T20:07:18.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:07:18.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:07:18.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=15 2026-02-22T20:07:18.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:07:18.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:07:18.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:07:18.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_data_pool /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.status 2026-02-22T20:07:18.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:18.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:18.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:18.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.status 2026-02-22T20:07:18.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_data_pool --format xml 2026-02-22T20:07:18.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:07:18.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:07:18.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=12 2026-02-22T20:07:18.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 15 = 12 2026-02-22T20:07:18.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:07:18.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:07:18.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_data_pool /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.status 2026-02-22T20:07:18.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:18.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:18.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:18.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.status 2026-02-22T20:07:18.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_data_pool --format xml 2026-02-22T20:07:18.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:07:18.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:07:18.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:07:18.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:07:18.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:07:18.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:07:18.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:07:18.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 15 = '' 2026-02-22T20:07:18.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:07:18.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.8 2026-02-22T20:07:19.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_data_pool /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.status 2026-02-22T20:07:19.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:19.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:19.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:19.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.status 2026-02-22T20:07:19.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_data_pool --format xml 2026-02-22T20:07:19.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:07:19.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:07:19.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:07:19.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:07:19.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:07:19.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:07:19.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:07:19.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 15 = '' 2026-02-22T20:07:19.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:07:19.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1.6 2026-02-22T20:07:21.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_data_pool /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.status 2026-02-22T20:07:21.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:21.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:21.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:21.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.status 2026-02-22T20:07:21.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_data_pool --format xml 2026-02-22T20:07:21.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:07:21.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:07:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=15 2026-02-22T20:07:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 15 = 15 2026-02-22T20:07:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_data_pool 2026-02-22T20:07:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:21.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:07:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_data_pool up+replaying 2026-02-22T20:07:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:07:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:07:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:07:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:07:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:07:22.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_data_pool up+replaying '' '' 2026-02-22T20:07:22.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:22.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:22.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:22.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:07:22.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:07:22.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:07:22.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:07:22.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:07:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_data_pool 2026-02-22T20:07:22.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_data_pool: 2026-02-22T20:07:22.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: bf0a2bc0-a879-4cc2-80f7-6ae2bcf3c1fd 2026-02-22T20:07:22.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:07:22.383 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790833,"replay_state":"idle"} 2026-02-22T20:07:22.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:07:22.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:07:15 2026-02-22T20:07:22.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:07:22.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:07:22.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:07:22.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:07:22.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:07:17' 2026-02-22T20:07:22.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:07:22.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:07:22.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:07:22.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790833,"replay_state":"idle"} 2026-02-22T20:07:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:07:22.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:07:22.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:07:22.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:07:22.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:07:22.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:07:22.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:07:22.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:07:22.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:07:15' 2026-02-22T20:07:22.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_data_pool up+replaying '2026-02-22 20:07:15' 2026-02-22T20:07:22.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:22.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:22.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:22.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:07:22.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:07:15' 2026-02-22T20:07:22.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:07:22.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:07:22.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:07:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK243384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:06:53cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:07:17test_data_poolbf0a2bc0-a879-4cc2-80f7-6ae2bcf3c1fdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790833,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 20:07:15cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:07:17' 2026-02-22T20:07:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:07:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/last_update' 2026-02-22T20:07:22.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:07:15' 2026-02-22T20:07:22.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/state' 2026-02-22T20:07:22.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:07:22.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:07:22.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:07:22.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:07:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test_data_pool@snap1 2026-02-22T20:07:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool@snap1 2026-02-22T20:07:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:07:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_data_pool@snap1.export 2026-02-22T20:07:22.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_data_pool@snap1.export 2026-02-22T20:07:22.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:07:22.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool@snap1.export 2026-02-22T20:07:22.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_data_pool@snap1.export 2026-02-22T20:07:22.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_data_pool@snap1.export 2026-02-22T20:07:22.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:07:22.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool@snap1.export 2026-02-22T20:07:22.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool@snap1.export 2026-02-22T20:07:22.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap1 /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool@snap1.export 2026-02-22T20:07:22.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:07:22.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap1 /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool@snap1.export 2026-02-22T20:07:22.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:07:22.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool@snap1.export 2026-02-22T20:07:22.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool@snap1.export 2026-02-22T20:07:22.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:22.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test_data_pool@snap2 2026-02-22T20:07:22.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:22.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool@snap2 2026-02-22T20:07:22.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:07:22.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_data_pool@snap2.export 2026-02-22T20:07:22.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_data_pool@snap2.export 2026-02-22T20:07:22.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:07:22.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool@snap2.export 2026-02-22T20:07:22.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_data_pool@snap2.export 2026-02-22T20:07:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_data_pool@snap2.export 2026-02-22T20:07:22.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:07:22.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool@snap2.export 2026-02-22T20:07:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool@snap2.export 2026-02-22T20:07:22.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap2 /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool@snap2.export 2026-02-22T20:07:22.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:07:22.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap2 /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool@snap2.export 2026-02-22T20:07:22.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:07:22.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool@snap2.export 2026-02-22T20:07:22.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool@snap2.export 2026-02-22T20:07:23.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:23.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test_data_pool 2026-02-22T20:07:23.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:23.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:23.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:07:23.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_data_pool.export 2026-02-22T20:07:23.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_data_pool.export 2026-02-22T20:07:23.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:07:23.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool.export 2026-02-22T20:07:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_data_pool.export 2026-02-22T20:07:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_data_pool.export 2026-02-22T20:07:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:07:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.export 2026-02-22T20:07:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.export 2026-02-22T20:07:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool.export 2026-02-22T20:07:23.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:07:23.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.export 2026-02-22T20:07:23.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:07:23.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.export 2026-02-22T20:07:23.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_data_pool.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_data_pool.export 2026-02-22T20:07:23.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test_data_pool 2026-02-22T20:07:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:07:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:07:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test_data_pool 2026-02-22T20:07:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:07:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_data_pool 2026-02-22T20:07:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2026-02-22T20:07:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test_data_pool 2026-02-22T20:07:27.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:07:27.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:27.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: disable mirroring / delete non-primary image' 2026-02-22T20:07:27.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:07:27.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:07:27.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:07:27 TEST: disable mirroring / delete non-primary image 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:07:27 TEST: disable mirroring / delete non-primary image 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image2=test2 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image3=test3 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image4=test4 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image5=test5 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test2 snapshot 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:27.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test2 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:27.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:07:27.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:07:27.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test2 2026-02-22T20:07:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:07:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test2 snapshot 2026-02-22T20:07:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test2 snapshot 2026-02-22T20:07:28.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:07:28.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test2 2026-02-22T20:07:28.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test2': 2026-02-22T20:07:28.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:07:28.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:07:28.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:07:28.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 15de289ed76f 2026-02-22T20:07:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.15de289ed76f 2026-02-22T20:07:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:07:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:07:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:07:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:07:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:07:27 2026 2026-02-22T20:07:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:07:27 2026 2026-02-22T20:07:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:07:27 2026 2026-02-22T20:07:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 15de289ed76f 2026-02-22T20:07:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:07:28.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:07:28.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 3f998622-b987-483f-8795-298d37fc5093 2026-02-22T20:07:28.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:07:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test2 100 2026-02-22T20:07:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:07:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:07:28.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:28.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:07:28.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:07:28.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:07:28.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:07:28.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.23 bytes/sec: 24 MiB/s 2026-02-22T20:07:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test2 snap1 2026-02-22T20:07:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:07:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap1 2026-02-22T20:07:29.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:07:29.156 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test2 snap2 2026-02-22T20:07:29.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:29.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:29.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:29.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap2 2026-02-22T20:07:29.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap2 2026-02-22T20:07:30.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:07:30.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' test2 = test4 ']' 2026-02-22T20:07:30.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' test2 = test5 ']' 2026-02-22T20:07:30.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test2 100 2026-02-22T20:07:30.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:30.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:30.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:30.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:07:30.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:07:30.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:30.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:07:30.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:07:30.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:07:30.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:07:30.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 5000.19 bytes/sec: 20 MiB/s 2026-02-22T20:07:30.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:07:30.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test2 2026-02-22T20:07:30.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:30.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:30.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:30.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test2 2026-02-22T20:07:31.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 157 2026-02-22T20:07:31.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test2 present 2026-02-22T20:07:31.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:31.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:31.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:31.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=present 2026-02-22T20:07:31.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:07:31.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:31.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:31.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test2 2026-02-22T20:07:31.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:07:31.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:07:31.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test2 2026-02-22T20:07:31.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-22T20:07:31.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:07:31.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id=110162bc5ac1 2026-02-22T20:07:31.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:07:31.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test2 present 110162bc5ac1 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=110162bc5ac1 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test2 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test2 2026-02-22T20:07:31.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-22T20:07:31.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:07:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=110162bc5ac1 2026-02-22T20:07:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 110162bc5ac1 2026-02-22T20:07:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 110162bc5ac1 -o 110162bc5ac1 = 110162bc5ac1 2026-02-22T20:07:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:07:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = present 2026-02-22T20:07:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snap_present cluster1 mirror test2 snap2 2026-02-22T20:07:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:31.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:31.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:31.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap2 2026-02-22T20:07:31.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:31.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:07:31.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:07:32.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2026-02-22T20:07:32.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:error setting snapshot context: (2) No such file or directory 2026-02-22T20:07:32.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ continue 2026-02-22T20:07:32.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:07:32.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:07:34.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2026-02-22T20:07:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test2': 2026-02-22T20:07:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:07:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:07:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 4 2026-02-22T20:07:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 110162bc5ac1 2026-02-22T20:07:34.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.110162bc5ac1 2026-02-22T20:07:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:07:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling, operations, non-primary 2026-02-22T20:07:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: snap-trash 2026-02-22T20:07:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:07:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:07:30 2026 2026-02-22T20:07:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:07:30 2026 2026-02-22T20:07:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:07:30 2026 2026-02-22T20:07:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: protected: False 2026-02-22T20:07:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 110162bc5ac1 2026-02-22T20:07:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:07:34.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:07:34.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 3f998622-b987-483f-8795-298d37fc5093 2026-02-22T20:07:34.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: false 2026-02-22T20:07:34.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:34.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-22T20:07:34.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test3 snapshot 2026-02-22T20:07:34.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:34.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:34.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:34.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:34.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:34.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:34.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test3 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:07:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test3 2026-02-22T20:07:34.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:34.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:34.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:07:34.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:34.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test3 snapshot 2026-02-22T20:07:34.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:34.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:34.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:34.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:34.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test3 snapshot 2026-02-22T20:07:35.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:07:35.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test3 2026-02-22T20:07:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test3': 2026-02-22T20:07:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:07:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:07:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:07:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 15f69f1f74fd 2026-02-22T20:07:35.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.15f69f1f74fd 2026-02-22T20:07:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:07:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:07:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:07:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:07:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:07:34 2026 2026-02-22T20:07:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:07:34 2026 2026-02-22T20:07:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:07:34 2026 2026-02-22T20:07:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 15f69f1f74fd 2026-02-22T20:07:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:07:35.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:07:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 6b1a6b5c-5d69-4097-90aa-eeafa3b1d0b8 2026-02-22T20:07:35.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:07:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test3 100 2026-02-22T20:07:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:07:35.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:07:35.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:35.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:07:35.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:07:35.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:07:35.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:07:35.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 8333.64 bytes/sec: 33 MiB/s 2026-02-22T20:07:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test3 snap1 2026-02-22T20:07:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:07:35.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap1 2026-02-22T20:07:36.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:07:36.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test3 snap2 2026-02-22T20:07:36.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:36.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:36.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:36.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap2 2026-02-22T20:07:36.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap2 2026-02-22T20:07:37.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' test3 = test4 ']' 2026-02-22T20:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' test3 = test5 ']' 2026-02-22T20:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test3 100 2026-02-22T20:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:07:37.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:07:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:07:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.23 bytes/sec: 24 MiB/s 2026-02-22T20:07:37.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:07:37.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test3 2026-02-22T20:07:37.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:37.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:37.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:37.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test3 2026-02-22T20:07:38.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 161 2026-02-22T20:07:38.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test3 present 2026-02-22T20:07:38.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:38.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:38.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:38.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=present 2026-02-22T20:07:38.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:07:38.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:38.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:38.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test3 2026-02-22T20:07:38.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:07:38.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:07:38.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test3 2026-02-22T20:07:38.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test3 2026-02-22T20:07:38.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:07:38.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id=110649e0ce99 2026-02-22T20:07:38.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:07:38.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test3 present 110649e0ce99 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=110649e0ce99 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test3 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test3 2026-02-22T20:07:38.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test3 2026-02-22T20:07:38.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=110649e0ce99 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 110649e0ce99 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 110649e0ce99 -o 110649e0ce99 = 110649e0ce99 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = present 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snap_present cluster1 mirror test3 snap2 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap2 2026-02-22T20:07:38.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:07:38.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:07:39.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror/test3@snap2 2026-02-22T20:07:39.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:error setting snapshot context: (2) No such file or directory 2026-02-22T20:07:39.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ continue 2026-02-22T20:07:39.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:07:39.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:07:41.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror/test3@snap2 2026-02-22T20:07:41.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test3': 2026-02-22T20:07:41.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:07:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:07:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 4 2026-02-22T20:07:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 110649e0ce99 2026-02-22T20:07:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.110649e0ce99 2026-02-22T20:07:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:07:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling, operations, non-primary 2026-02-22T20:07:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: snap-trash 2026-02-22T20:07:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:07:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:07:37 2026 2026-02-22T20:07:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:07:37 2026 2026-02-22T20:07:41.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:07:37 2026 2026-02-22T20:07:41.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: protected: False 2026-02-22T20:07:41.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 110649e0ce99 2026-02-22T20:07:41.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:07:41.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:07:41.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 6b1a6b5c-5d69-4097-90aa-eeafa3b1d0b8 2026-02-22T20:07:41.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: false 2026-02-22T20:07:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-22T20:07:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test4 snapshot 2026-02-22T20:07:41.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test4 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:07:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:07:41.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test4 2026-02-22T20:07:41.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:41.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:41.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:07:41.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:41.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test4 snapshot 2026-02-22T20:07:41.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:41.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:41.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:41.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:41.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test4 snapshot 2026-02-22T20:07:41.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:07:41.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test4 2026-02-22T20:07:41.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test4': 2026-02-22T20:07:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:07:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:07:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:07:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 160ee79f65af 2026-02-22T20:07:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.160ee79f65af 2026-02-22T20:07:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:07:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:07:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:07:41.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:07:41 2026 2026-02-22T20:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:07:41 2026 2026-02-22T20:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:07:41 2026 2026-02-22T20:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 160ee79f65af 2026-02-22T20:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: d3a9cc18-4cc4-41fa-a8a0-01185a4533a6 2026-02-22T20:07:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:07:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test4 100 2026-02-22T20:07:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:07:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:07:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:41.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:07:41.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:07:42.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:07:42.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:07:42.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.23 bytes/sec: 24 MiB/s 2026-02-22T20:07:42.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test4 snap1 2026-02-22T20:07:42.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:42.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:42.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:42.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:07:42.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap1 2026-02-22T20:07:42.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:07:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test4 snap2 2026-02-22T20:07:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap2 2026-02-22T20:07:42.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap2 2026-02-22T20:07:43.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:07:43.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' test4 = test4 ']' 2026-02-22T20:07:43.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ protect_snapshot cluster2 mirror test4 snap1 2026-02-22T20:07:43.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:43.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:43.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:43.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:07:43.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap1 2026-02-22T20:07:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ protect_snapshot cluster2 mirror test4 snap2 2026-02-22T20:07:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap2 2026-02-22T20:07:43.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap2 2026-02-22T20:07:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test4 100 2026-02-22T20:07:43.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:07:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:07:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:07:43.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:07:43.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:07:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:07:43.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.23 bytes/sec: 24 MiB/s 2026-02-22T20:07:43.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:07:43.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test4 2026-02-22T20:07:43.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:43.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:43.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:43.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test4 2026-02-22T20:07:44.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 165 2026-02-22T20:07:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test4 present 2026-02-22T20:07:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=present 2026-02-22T20:07:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:07:44.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:44.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:44.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test4 2026-02-22T20:07:44.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:07:44.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:07:44.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test4 2026-02-22T20:07:44.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-22T20:07:44.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:07:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id=1109a6375aa9 2026-02-22T20:07:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:07:44.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:07:44.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test4 present 1109a6375aa9 2026-02-22T20:07:44.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:44.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:44.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:44.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:07:44.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=1109a6375aa9 2026-02-22T20:07:44.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:07:44.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:07:44.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test4 2026-02-22T20:07:44.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:07:44.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:07:44.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test4 2026-02-22T20:07:44.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-22T20:07:44.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:07:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=1109a6375aa9 2026-02-22T20:07:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 1109a6375aa9 2026-02-22T20:07:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 1109a6375aa9 -o 1109a6375aa9 = 1109a6375aa9 2026-02-22T20:07:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:07:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = present 2026-02-22T20:07:44.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:44.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snap_present cluster1 mirror test4 snap2 2026-02-22T20:07:44.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:44.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:44.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:44.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap2 2026-02-22T20:07:44.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:44.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:07:44.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:07:45.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2026-02-22T20:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:error setting snapshot context: (2) No such file or directory 2026-02-22T20:07:45.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ continue 2026-02-22T20:07:45.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:07:45.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:07:47.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2026-02-22T20:07:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test4': 2026-02-22T20:07:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:07:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:07:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 3 2026-02-22T20:07:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 1109a6375aa9 2026-02-22T20:07:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.1109a6375aa9 2026-02-22T20:07:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:07:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling, non-primary 2026-02-22T20:07:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:07:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:07:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:07:43 2026 2026-02-22T20:07:47.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:07:43 2026 2026-02-22T20:07:47.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:07:43 2026 2026-02-22T20:07:47.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: protected: True 2026-02-22T20:07:47.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 1109a6375aa9 2026-02-22T20:07:47.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:07:47.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:07:47.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: d3a9cc18-4cc4-41fa-a8a0-01185a4533a6 2026-02-22T20:07:47.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: false 2026-02-22T20:07:47.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:47.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-22T20:07:47.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test5 snapshot 2026-02-22T20:07:47.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:47.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:47.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:47.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:47.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test5 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:07:47.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test5 2026-02-22T20:07:47.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:07:47.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:47.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:07:47.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:47.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test5 snapshot 2026-02-22T20:07:47.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:47.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:47.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:47.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:07:47.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test5 snapshot 2026-02-22T20:07:48.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:07:48.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test5 2026-02-22T20:07:48.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test5': 2026-02-22T20:07:48.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:07:48.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:07:48.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:07:48.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 162c558a0991 2026-02-22T20:07:48.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.162c558a0991 2026-02-22T20:07:48.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:07:48.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:07:48.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:07:48.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:07:48.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:07:47 2026 2026-02-22T20:07:48.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:07:47 2026 2026-02-22T20:07:48.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:07:47 2026 2026-02-22T20:07:48.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 162c558a0991 2026-02-22T20:07:48.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:07:48.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:07:48.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 07e36a41-ed71-4748-878b-8b27ce54f916 2026-02-22T20:07:48.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:07:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test5 100 2026-02-22T20:07:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:07:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:07:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:07:48.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:07:48.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:07:48.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:07:48.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.23 bytes/sec: 24 MiB/s 2026-02-22T20:07:48.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test5 snap1 2026-02-22T20:07:48.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:48.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:48.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:48.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:07:48.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap1 2026-02-22T20:07:49.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:07:49.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test5 snap2 2026-02-22T20:07:49.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:49.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:49.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:49.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap2 2026-02-22T20:07:49.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap2 2026-02-22T20:07:50.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:07:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' test5 = test4 ']' 2026-02-22T20:07:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' test5 = test5 ']' 2026-02-22T20:07:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ protect_snapshot cluster2 mirror test5 snap1 2026-02-22T20:07:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:50.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:50.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:07:50.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap1 2026-02-22T20:07:50.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ protect_snapshot cluster2 mirror test5 snap2 2026-02-22T20:07:50.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:50.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:50.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:50.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap2 2026-02-22T20:07:50.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap2 2026-02-22T20:07:50.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test5 100 2026-02-22T20:07:50.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:50.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:50.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:50.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:07:50.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:07:50.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:50.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:07:50.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:07:50.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:07:50.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:07:50.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 5000.18 bytes/sec: 20 MiB/s 2026-02-22T20:07:50.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:07:50.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test5 2026-02-22T20:07:50.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:50.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:50.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:50.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test5 2026-02-22T20:07:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 169 2026-02-22T20:07:51.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test5 present 2026-02-22T20:07:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=present 2026-02-22T20:07:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:07:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:07:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test5 2026-02-22T20:07:51.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:07:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:07:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test5 2026-02-22T20:07:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test5 2026-02-22T20:07:51.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:07:51.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id=10fa198c6394 2026-02-22T20:07:51.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:07:51.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test5 present 10fa198c6394 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=10fa198c6394 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test5 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:07:51.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test5 2026-02-22T20:07:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test5 2026-02-22T20:07:51.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:07:51.335 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=10fa198c6394 2026-02-22T20:07:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 10fa198c6394 2026-02-22T20:07:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 10fa198c6394 -o 10fa198c6394 = 10fa198c6394 2026-02-22T20:07:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:07:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = present 2026-02-22T20:07:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snap_present cluster1 mirror test5 snap2 2026-02-22T20:07:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:07:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:51.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap2 2026-02-22T20:07:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:07:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:07:51.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:07:52.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror/test5@snap2 2026-02-22T20:07:52.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:error setting snapshot context: (2) No such file or directory 2026-02-22T20:07:52.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ continue 2026-02-22T20:07:52.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:07:52.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:07:54.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror/test5@snap2 2026-02-22T20:07:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test5': 2026-02-22T20:07:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:07:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:07:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 4 2026-02-22T20:07:54.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 10fa198c6394 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.10fa198c6394 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling, operations, non-primary 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: snap-trash 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:07:50 2026 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:07:50 2026 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:07:50 2026 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: protected: True 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 10fa198c6394 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:07:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:07:54.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 07e36a41-ed71-4748-878b-8b27ce54f916 2026-02-22T20:07:54.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: false 2026-02-22T20:07:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:07:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_pool_mirror_mode cluster2 mirror image 2026-02-22T20:07:54.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=image 2026-02-22T20:07:54.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2026-02-22T20:07:54.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd: mirroring is already configured for image mode 2026-02-22T20:07:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image4} 2026-02-22T20:07:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ disable_mirror cluster2 mirror test2 2026-02-22T20:07:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:07:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test2 2026-02-22T20:07:56.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring disabled 2026-02-22T20:07:56.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image4} 2026-02-22T20:07:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ disable_mirror cluster2 mirror test4 2026-02-22T20:07:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:07:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test4 2026-02-22T20:07:58.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring disabled 2026-02-22T20:07:58.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ unprotect_snapshot cluster2 mirror test5 snap1 2026-02-22T20:07:58.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:58.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:58.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:58.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:07:58.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap1 2026-02-22T20:07:58.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ unprotect_snapshot cluster2 mirror test5 snap2 2026-02-22T20:07:58.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:58.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:58.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:07:58.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap2 2026-02-22T20:07:58.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap2 2026-02-22T20:07:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image3} ${image5} 2026-02-22T20:07:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_snapshot cluster2 mirror test3 snap1 2026-02-22T20:07:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:07:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap1 2026-02-22T20:07:59.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing snap: 100% complete...done. 2026-02-22T20:07:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_snapshot cluster2 mirror test3 snap2 2026-02-22T20:07:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:07:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:07:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:07:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap2 2026-02-22T20:07:59.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap2 2026-02-22T20:08:00.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing snap: 100% complete...done. 2026-02-22T20:08:00.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test3 2026-02-22T20:08:00.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:00.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:00.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:08:00.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:00.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:08:00.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test3 2026-02-22T20:08:00.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:00.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:00.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:08:00.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test3 2026-02-22T20:08:00.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test3 2026-02-22T20:08:02.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:08:02.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:02.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image3} ${image5} 2026-02-22T20:08:02.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_snapshot cluster2 mirror test5 snap1 2026-02-22T20:08:02.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:02.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:02.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:08:02.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:08:02.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap1 2026-02-22T20:08:03.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing snap: 100% complete...done. 2026-02-22T20:08:03.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_snapshot cluster2 mirror test5 snap2 2026-02-22T20:08:03.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:03.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:03.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:08:03.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap2 2026-02-22T20:08:03.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap2 2026-02-22T20:08:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing snap: 100% complete...done. 2026-02-22T20:08:04.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test5 2026-02-22T20:08:04.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:04.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:04.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:08:04.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:04.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:08:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test5 2026-02-22T20:08:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:08:04.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test5 2026-02-22T20:08:04.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test5 2026-02-22T20:08:06.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:08:06.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-22T20:08:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test2 deleted 2026-02-22T20:08:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T20:08:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:08:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:08:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test2 2026-02-22T20:08:06.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:08:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test2 2026-02-22T20:08:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-22T20:08:06.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:08:06.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test2: (2) No such file or directory 2026-02-22T20:08:06.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id= 2026-02-22T20:08:06.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:08:06.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:08:06.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test2 deleted '' 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test2 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test2 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test2 2026-02-22T20:08:06.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:08:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test2: (2) No such file or directory 2026-02-22T20:08:06.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:08:06.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:06.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T20:08:06.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:06.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-22T20:08:06.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test3 deleted 2026-02-22T20:08:06.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:06.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:06.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:08:06.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T20:08:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:08:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:08:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test3 2026-02-22T20:08:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:08:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test3 2026-02-22T20:08:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test3 2026-02-22T20:08:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:08:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test3: (2) No such file or directory 2026-02-22T20:08:06.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id= 2026-02-22T20:08:06.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:08:06.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:08:06.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test3 deleted '' 2026-02-22T20:08:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test3 2026-02-22T20:08:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:08:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:08:06.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:08:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:08:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test3 2026-02-22T20:08:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:08:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test3 2026-02-22T20:08:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test3 2026-02-22T20:08:06.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:08:06.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test3: (2) No such file or directory 2026-02-22T20:08:06.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test4 deleted 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:06.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test4 2026-02-22T20:08:06.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:06.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:08:06.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test4 2026-02-22T20:08:06.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-22T20:08:06.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:08:06.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test4: (2) No such file or directory 2026-02-22T20:08:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id= 2026-02-22T20:08:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:08:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test4 deleted '' 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test4 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test4 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test4 2026-02-22T20:08:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:08:06.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test4: (2) No such file or directory 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test5 deleted 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test5 2026-02-22T20:08:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:08:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test5 2026-02-22T20:08:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test5 2026-02-22T20:08:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:08:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test5: (2) No such file or directory 2026-02-22T20:08:06.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id= 2026-02-22T20:08:06.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:08:06.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test5 deleted '' 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test5 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test5 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:08:06.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test5 2026-02-22T20:08:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test5 2026-02-22T20:08:06.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:08:06.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test5: (2) No such file or directory 2026-02-22T20:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T20:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: snapshot rename' 2026-02-22T20:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:08:06 TEST: snapshot rename 2026-02-22T20:08:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:08:06 TEST: snapshot rename 2026-02-22T20:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ snap_name=snap_rename 2026-02-22T20:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test2 2026-02-22T20:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:08:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test2 snapshot 2026-02-22T20:08:07.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:08:07.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test2 2026-02-22T20:08:07.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test2': 2026-02-22T20:08:07.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:08:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:08:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 3 2026-02-22T20:08:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 15de289ed76f 2026-02-22T20:08:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.15de289ed76f 2026-02-22T20:08:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:08:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:08:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:08:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:08:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:07:27 2026 2026-02-22T20:08:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:07:27 2026 2026-02-22T20:08:07.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:07:27 2026 2026-02-22T20:08:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 15de289ed76f 2026-02-22T20:08:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:08:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:08:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 69332554-7e5a-47a8-a85d-1c808873b228 2026-02-22T20:08:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:08:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test2 snap_rename_0 2026-02-22T20:08:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_0 2026-02-22T20:08:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap_rename_0 2026-02-22T20:08:08.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:08:08.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 1 20 2026-02-22T20:08:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 1 - 1 2026-02-22T20:08:08.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_0 snap_rename_1 2026-02-22T20:08:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_0 2026-02-22T20:08:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_1 2026-02-22T20:08:08.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_0 mirror/test2@snap_rename_1 2026-02-22T20:08:08.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 2 - 1 2026-02-22T20:08:08.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_1 snap_rename_2 2026-02-22T20:08:08.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_1 2026-02-22T20:08:08.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_2 2026-02-22T20:08:08.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_1 mirror/test2@snap_rename_2 2026-02-22T20:08:08.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 3 - 1 2026-02-22T20:08:08.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_2 snap_rename_3 2026-02-22T20:08:08.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_2 2026-02-22T20:08:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_3 2026-02-22T20:08:08.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_2 mirror/test2@snap_rename_3 2026-02-22T20:08:08.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 4 - 1 2026-02-22T20:08:08.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_3 snap_rename_4 2026-02-22T20:08:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_3 2026-02-22T20:08:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_4 2026-02-22T20:08:08.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_3 mirror/test2@snap_rename_4 2026-02-22T20:08:08.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 5 - 1 2026-02-22T20:08:08.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_4 snap_rename_5 2026-02-22T20:08:08.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_4 2026-02-22T20:08:08.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_5 2026-02-22T20:08:08.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_4 mirror/test2@snap_rename_5 2026-02-22T20:08:08.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 6 - 1 2026-02-22T20:08:08.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_5 snap_rename_6 2026-02-22T20:08:08.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_5 2026-02-22T20:08:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_6 2026-02-22T20:08:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_5 mirror/test2@snap_rename_6 2026-02-22T20:08:08.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 7 - 1 2026-02-22T20:08:08.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_6 snap_rename_7 2026-02-22T20:08:08.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_6 2026-02-22T20:08:08.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_7 2026-02-22T20:08:08.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_6 mirror/test2@snap_rename_7 2026-02-22T20:08:08.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 8 - 1 2026-02-22T20:08:08.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_7 snap_rename_8 2026-02-22T20:08:08.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_7 2026-02-22T20:08:08.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_8 2026-02-22T20:08:08.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_7 mirror/test2@snap_rename_8 2026-02-22T20:08:08.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 9 - 1 2026-02-22T20:08:08.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_8 snap_rename_9 2026-02-22T20:08:08.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_8 2026-02-22T20:08:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_9 2026-02-22T20:08:08.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_8 mirror/test2@snap_rename_9 2026-02-22T20:08:08.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 10 - 1 2026-02-22T20:08:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_9 snap_rename_10 2026-02-22T20:08:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_9 2026-02-22T20:08:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_10 2026-02-22T20:08:08.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_9 mirror/test2@snap_rename_10 2026-02-22T20:08:08.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 11 - 1 2026-02-22T20:08:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_10 snap_rename_11 2026-02-22T20:08:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_10 2026-02-22T20:08:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_11 2026-02-22T20:08:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_10 mirror/test2@snap_rename_11 2026-02-22T20:08:08.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 12 - 1 2026-02-22T20:08:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_11 snap_rename_12 2026-02-22T20:08:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_11 2026-02-22T20:08:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_12 2026-02-22T20:08:08.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_11 mirror/test2@snap_rename_12 2026-02-22T20:08:08.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 13 - 1 2026-02-22T20:08:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_12 snap_rename_13 2026-02-22T20:08:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_12 2026-02-22T20:08:08.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_13 2026-02-22T20:08:08.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_12 mirror/test2@snap_rename_13 2026-02-22T20:08:08.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:08.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 14 - 1 2026-02-22T20:08:08.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_13 snap_rename_14 2026-02-22T20:08:08.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:08.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:08.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:08.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_13 2026-02-22T20:08:08.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_14 2026-02-22T20:08:08.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_13 mirror/test2@snap_rename_14 2026-02-22T20:08:09.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:09.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 15 - 1 2026-02-22T20:08:09.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_14 snap_rename_15 2026-02-22T20:08:09.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:09.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:09.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:09.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_14 2026-02-22T20:08:09.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_15 2026-02-22T20:08:09.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_14 mirror/test2@snap_rename_15 2026-02-22T20:08:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 16 - 1 2026-02-22T20:08:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_15 snap_rename_16 2026-02-22T20:08:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_15 2026-02-22T20:08:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_16 2026-02-22T20:08:09.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_15 mirror/test2@snap_rename_16 2026-02-22T20:08:09.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:09.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 17 - 1 2026-02-22T20:08:09.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_16 snap_rename_17 2026-02-22T20:08:09.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:09.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:09.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:09.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_16 2026-02-22T20:08:09.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_17 2026-02-22T20:08:09.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_16 mirror/test2@snap_rename_17 2026-02-22T20:08:09.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:09.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 18 - 1 2026-02-22T20:08:09.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_17 snap_rename_18 2026-02-22T20:08:09.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:09.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:09.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:09.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_17 2026-02-22T20:08:09.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_18 2026-02-22T20:08:09.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_17 mirror/test2@snap_rename_18 2026-02-22T20:08:09.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:09.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 19 - 1 2026-02-22T20:08:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_18 snap_rename_19 2026-02-22T20:08:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:09.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_18 2026-02-22T20:08:09.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_19 2026-02-22T20:08:09.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_18 mirror/test2@snap_rename_19 2026-02-22T20:08:09.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in `seq 1 20` 2026-02-22T20:08:09.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ expr 20 - 1 2026-02-22T20:08:09.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_19 snap_rename_20 2026-02-22T20:08:09.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:09.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:09.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:09.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap_rename_19 2026-02-22T20:08:09.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local new_snap=snap_rename_20 2026-02-22T20:08:09.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_19 mirror/test2@snap_rename_20 2026-02-22T20:08:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:08:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test2 2026-02-22T20:08:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test2 2026-02-22T20:08:10.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 172 2026-02-22T20:08:10.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snap_present cluster1 mirror test2 snap_rename_20 2026-02-22T20:08:10.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:10.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:10.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:10.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap_rename_20 2026-02-22T20:08:10.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:08:10.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:08:10.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:11.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2026-02-22T20:08:11.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:error setting snapshot context: (2) No such file or directory 2026-02-22T20:08:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ continue 2026-02-22T20:08:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:08:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:08:13.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2026-02-22T20:08:13.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test2': 2026-02-22T20:08:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:08:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:08:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 4 2026-02-22T20:08:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 1101a737433 2026-02-22T20:08:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.1101a737433 2026-02-22T20:08:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:08:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling, non-primary 2026-02-22T20:08:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:08:13.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:08:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:08:09 2026 2026-02-22T20:08:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:08:09 2026 2026-02-22T20:08:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:08:09 2026 2026-02-22T20:08:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: protected: False 2026-02-22T20:08:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 1101a737433 2026-02-22T20:08:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:08:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:08:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 69332554-7e5a-47a8-a85d-1c808873b228 2026-02-22T20:08:13.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: false 2026-02-22T20:08:13.271 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ unprotect_snapshot cluster2 mirror test4 snap1 2026-02-22T20:08:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:08:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:08:13.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test4@snap1 2026-02-22T20:08:13.320 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ unprotect_snapshot cluster2 mirror test4 snap2 2026-02-22T20:08:13.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:13.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:13.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:08:13.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap2 2026-02-22T20:08:13.321 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test4@snap2 2026-02-22T20:08:13.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image4} 2026-02-22T20:08:13.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test2 2026-02-22T20:08:13.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:13.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:13.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:13.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:13.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:08:13.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test2 2026-02-22T20:08:13.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:13.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:13.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test2 2026-02-22T20:08:13.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test2 2026-02-22T20:08:16.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing all snapshots: 33% complete... Removing all snapshots: 66% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2026-02-22T20:08:16.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test2 2026-02-22T20:08:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:08:18.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:18.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for i in ${image2} ${image4} 2026-02-22T20:08:18.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test4 2026-02-22T20:08:18.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:18.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:18.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:08:18.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:18.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:08:18.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test4 2026-02-22T20:08:18.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:18.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:18.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test4 2026-02-22T20:08:18.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test4 2026-02-22T20:08:20.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2026-02-22T20:08:20.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test4 2026-02-22T20:08:20.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:08:20.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:20.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z '' ']' 2026-02-22T20:08:20.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: demote image while daemon is offline' 2026-02-22T20:08:20.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:08:20.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:08:20.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:08:20 TEST: demote image while daemon is offline 2026-02-22T20:08:20.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:08:20 TEST: demote image while daemon is offline 2026-02-22T20:08:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image=test_demote_image 2026-02-22T20:08:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test_demote_image snapshot 2026-02-22T20:08:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test_demote_image 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:20.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:20.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:20.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:08:20.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:08:20.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test_demote_image 2026-02-22T20:08:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:08:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:08:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test_demote_image snapshot 2026-02-22T20:08:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:20.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:20.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:08:20.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test_demote_image snapshot 2026-02-22T20:08:21.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:08:21.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test_demote_image 2026-02-22T20:08:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_demote_image': 2026-02-22T20:08:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:08:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:08:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 16ca942d6e25 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.16ca942d6e25 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:08:20 2026 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:08:20 2026 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:08:20 2026 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 16ca942d6e25 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 22f785c3-8fce-4561-b0d3-a0dffd391329 2026-02-22T20:08:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:08:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_demote_image 100 2026-02-22T20:08:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:08:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:08:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:08:21.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_demote_image --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:08:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:08:21.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:08:21.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.21 bytes/sec: 24 MiB/s 2026-02-22T20:08:21.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_demote_image 2026-02-22T20:08:21.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:21.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:21.259 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:21.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_demote_image stopped 2026-02-22T20:08:21.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:21.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:21.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:21.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:08:21.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:08:21.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:21.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_demote_image stopped 2026-02-22T20:08:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:22.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:08:22.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:08:22.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:08:22.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:22.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:08:22.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:08:22.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:22.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:08:22.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:08:22.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:08:22.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:22.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:08:22.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:08:22.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:08:22.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:22.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:22.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:08:22.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:22.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:22.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:08:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:08:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:08:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:08:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:08:22.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:08:22.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:22.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:22.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:22.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:08:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-22T20:08:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:22.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:22.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:22.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:08:22.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:08:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:08:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:08:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:08:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:22.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:22.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:08:22.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:22.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:22.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:08:22.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:08:22.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:08:22.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:22.340 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:22.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:22.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:08:22.398 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-22T20:08:22.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:22.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:22.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:22.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:2 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:22.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:2 2026-02-22T20:08:22.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:22.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:22.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-22T20:08:22.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:08:22.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:22.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:22.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:22.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:08:22.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:2 cluster2 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:2 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:2 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:22.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:22.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:22.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:08:22.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-22T20:08:22.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:22.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:22.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:08:22.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:08:22.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-22T20:08:22.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:08:22.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:08:22.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:08:22.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:08:22.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-22T20:08:22.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:22.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:22.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:22.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:3 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:22.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:3 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:3 cluster2 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:3 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:3 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:08:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-22T20:08:22.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:22.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:22.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:08:22.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:08:22.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-22T20:08:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:08:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:08:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:08:22.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:08:22.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-22T20:08:22.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:22.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:22.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:22.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T20:08:23.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:08:23.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:08:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:08:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:08:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:08:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:08:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:08:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:23.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:08:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:08:23.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:08:23.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:23.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:23.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:23.550 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:08:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-22T20:08:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:23.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:23.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:23.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:08:23.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:23.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:23.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:08:23.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:08:23.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:23.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:23.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:08:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:08:23.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:23.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:23.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:23.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:23.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:08:23.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:23.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:23.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:08:23.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:08:23.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:08:23.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:23.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:23.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:23.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:08:23.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:08:23.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:08:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:08:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:08:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:08:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_demote_image 2026-02-22T20:08:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_demote_image started 2026-02-22T20:08:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:08:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:08:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:23.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:24.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_demote_image started 2026-02-22T20:08:24.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:24.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:24.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:24.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:08:24.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:08:24.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:08:24.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:24.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:08:24.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:08:24.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:24.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:24.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:08:24.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:08:24.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:24.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:08:24.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:08:24.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:08:24.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:24.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:24.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:08:24.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:24.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:24.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:08:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:08:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:08:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:08:24.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:08:24.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:24.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:08:24.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:24.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:08:24.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:08:24.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:08:24.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:24.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:08:24.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:08:24.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:24.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:24.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:24.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:24.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:08:24.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:08:24.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:08:24.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:08:24.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:08:24.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:08:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:08:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:08:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:08:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:08:24.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-22T20:08:24.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:24.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:24.753 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:24.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:24.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:08:24.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:24.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:24.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:08:24.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:08:24.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:24.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:24.754 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:24.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:08:24.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:08:24.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:08:24.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:24.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:08:24.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:24.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:08:24.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:08:24.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:08:24.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:24.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:08:24.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:08:24.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:24.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:24.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:24.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:24.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:08:24.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:08:24.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:08:24.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:08:24.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:08:24.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:08:24.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:08:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:08:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:08:24.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:08:24.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:08:24.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:08:24.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:08:24.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:08:24.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:08:24.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:08:24.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:08:24.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:24.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_demote_image 2026-02-22T20:08:24.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:08:24.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:24.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:24.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:24.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:08:24.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:08:24.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test_demote_image 2026-02-22T20:08:24.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:24.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:24.831 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:24.832 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test_demote_image 2026-02-22T20:08:25.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 174 2026-02-22T20:08:25.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test_demote_image 2026-02-22T20:08:25.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:08:25.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:25.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:25.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:25.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_demote_image.status 2026-02-22T20:08:25.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_demote_image.status 2026-02-22T20:08:25.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:08:25.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:08:25.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_demote_image.status 2026-02-22T20:08:25.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_demote_image.status 2026-02-22T20:08:25.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:08:25.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:25.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:08:25.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:25.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:25.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:25.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:08:25.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:08:25.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:25.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=174 2026-02-22T20:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:08:25.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:08:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:08:25.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:25.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:08:25.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=173 2026-02-22T20:08:25.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 174 = 173 2026-02-22T20:08:25.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:08:25.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:08:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:08:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:25.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:08:25.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=174 2026-02-22T20:08:25.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 174 = 174 2026-02-22T20:08:25.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:25.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_demote_image up+replaying 2026-02-22T20:08:25.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:25.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:25.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:25.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:08:25.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:25.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:25.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:25.897 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:26.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+replaying '' '' 2026-02-22T20:08:26.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:26.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:26.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:26.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:08:26.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:26.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:26.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:26.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:26.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-22T20:08:26.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:08:26.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 22f785c3-8fce-4561-b0d3-a0dffd391329 2026-02-22T20:08:26.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:08:26.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790901,"replay_state":"idle"} 2026-02-22T20:08:26.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:08:26.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:23 2026-02-22T20:08:26.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:08:26.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:08:26.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:08:26.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:08:26.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:08:26.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:08:26.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:08:26.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:08:26.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790901,"replay_state":"idle"} 2026-02-22T20:08:26.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:08:26.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:08:26.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:08:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:08:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:08:26.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:08:26.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:08:26.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:08:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:23' 2026-02-22T20:08:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_demote_image up+replaying '2026-02-22 20:08:23' 2026-02-22T20:08:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:08:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:08:23' 2026-02-22T20:08:26.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:08:26.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING1143384346mirror.0trial01719.2.3-689-gf90f4eaetrueOK43424353mirror.1trial01719.2.3-689-gf90f4eaefalseOK43484358mirror.2trial01719.2.3-689-gf90f4eaefalseOK43554361mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}43384346mirror.0trial0172026-02-22 20:08:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:08:22test_demote_image22f785c3-8fce-4561-b0d3-a0dffd391329up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790901,"replay_state":"idle"}43424353mirror.1trial0172026-02-22 20:08:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:08:26.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:08:26.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:08:26.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:23' 2026-02-22T20:08:26.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:08:26.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:08:26.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:08:26.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:08:26.995 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:08:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_demote_image up+stopped 2026-02-22T20:08:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:26.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:08:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:26.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:27.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_demote_image up+stopped '' '' 2026-02-22T20:08:27.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:27.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:27.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:27.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:08:27.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:27.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:27.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:27.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:27.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_demote_image 2026-02-22T20:08:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:08:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 22f785c3-8fce-4561-b0d3-a0dffd391329 2026-02-22T20:08:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:08:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:08:28.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:08:28.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:27 2026-02-22T20:08:28.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:08:28.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:08:28.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:08:28.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790901,"replay_state":"idle"} 2026-02-22T20:08:28.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:23 2026-02-22T20:08:28.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:08:28.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 173 .mirror.primary.22f785c3-8fce-4561-b0d3-a0dffd391329.4baeb5ae-2480-4f08-971e-d01819899b1d (peer_uuids:[]) 2026-02-22T20:08:28.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 174 .mirror.primary.22f785c3-8fce-4561-b0d3-a0dffd391329.6cee09c5-1b77-49fe-ac8f-c40731f9c4a1 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:08:28.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:08:28.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:08:28.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:08:28.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:08:28.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:08:28.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:08:28.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:08:28.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:08:28.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:08:28.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:08:28.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:08:28.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:08:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:27' 2026-02-22T20:08:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_demote_image up+stopped '2026-02-22 20:08:27' 2026-02-22T20:08:28.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:08:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:08:27' 2026-02-22T20:08:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:28.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:08:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK244524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:08:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}2026-02-22 20:08:23test_demote_image22f785c3-8fce-4561-b0d3-a0dffd391329up+stoppedlocal image is primary44564466mirror.1trial0172026-02-22 20:08:27cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790901,"replay_state":"idle"}2026-02-22 20:08:23' 2026-02-22T20:08:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:08:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:08:28.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:27' 2026-02-22T20:08:28.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:08:28.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:08:28.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:08:28.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:08:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:08:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirrors cluster1 2026-02-22T20:08:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:08:28.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:08:28.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:0 '' 2026-02-22T20:08:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:08:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:08:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:08:28.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:0 2026-02-22T20:08:28.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:0 2026-02-22T20:08:28.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:08:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:08:28.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:08:28.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:08:28.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=21766 2026-02-22T20:08:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 21766 ']' 2026-02-22T20:08:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 21766 2026-02-22T20:08:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:28.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:08:28.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:28.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21766 '$2 == pid {print; exit 1}' 2026-02-22T20:08:28.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 21766 0.3 0.0 775136 59932 ? Ssl 19:53 0:03 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:08:28.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:28.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21766 '$2 == pid {print; exit 1}' 2026-02-22T20:08:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:08:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:29.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21766 '$2 == pid {print; exit 1}' 2026-02-22T20:08:29.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:08:29.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:08:29.140 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:29.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:08:29.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:08:29.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:29.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:08:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:08:29.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:08:29.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:0 2026-02-22T20:08:29.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:08:29.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:29.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:08:29.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:08:29.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:29.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:29.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:29.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:08:29.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:08:29.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:08:29.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:29.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:08:29.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:29.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:08:29.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:08:29.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:08:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:1 '' 2026-02-22T20:08:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:08:29.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:08:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:1 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:1 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:08:29.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:08:29.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:08:29.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:08:29.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:08:29.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:08:29.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:08:29.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=21794 2026-02-22T20:08:29.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 21794 ']' 2026-02-22T20:08:29.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 21794 2026-02-22T20:08:29.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:29.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:08:29.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:29.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21794 '$2 == pid {print; exit 1}' 2026-02-22T20:08:29.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 21794 0.2 0.0 848896 58644 ? Rsl 19:53 0:02 rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:08:29.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:29.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:30.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:30.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21794 '$2 == pid {print; exit 1}' 2026-02-22T20:08:30.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:08:30.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:30.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21794 '$2 == pid {print; exit 1}' 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:30.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:08:30.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:08:30.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:08:30.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:30.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:08:30.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:08:30.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:1 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:08:30.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:30.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:08:30.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:08:30.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:08:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:30.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:2 '' 2026-02-22T20:08:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:08:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:08:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:30.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:08:30.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:2 2026-02-22T20:08:30.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:2 2026-02-22T20:08:30.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:30.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:08:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:08:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:08:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:08:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:08:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:08:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:08:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:08:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:08:30.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:08:30.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:08:30.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:08:30.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=21837 2026-02-22T20:08:30.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 21837 ']' 2026-02-22T20:08:30.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 21837 2026-02-22T20:08:30.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:30.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:08:30.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:30.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21837 '$2 == pid {print; exit 1}' 2026-02-22T20:08:30.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 21837 0.1 0.0 684980 54048 ? Rsl 19:53 0:01 rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:08:30.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:30.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:31.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:31.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21837 '$2 == pid {print; exit 1}' 2026-02-22T20:08:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:08:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:31.325 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21837 '$2 == pid {print; exit 1}' 2026-02-22T20:08:31.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:08:31.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:08:31.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:31.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:31.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:08:31.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:08:31.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:08:31.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:31.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:31.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:08:31.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:08:31.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:08:31.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:08:31.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:08:31.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:08:31.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:08:31.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:08:31.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:08:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-22T20:08:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:08:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:08:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:31.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:08:31.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:08:31.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:08:31.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:31.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:31.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:08:31.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:08:31.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:08:31.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:08:31.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:08:31.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:08:31.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:08:31.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:08:31.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:08:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:2 2026-02-22T20:08:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:08:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:08:31.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:08:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:08:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:08:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:08:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:08:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:08:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:08:31.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:08:31.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:08:31.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:31.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:3 '' 2026-02-22T20:08:31.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:08:31.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:08:31.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:31.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:08:31.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:3 2026-02-22T20:08:31.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:3 2026-02-22T20:08:31.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:08:31.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:08:31.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:08:31.390 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:08:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=21889 2026-02-22T20:08:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 21889 ']' 2026-02-22T20:08:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 21889 2026-02-22T20:08:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:31.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:08:31.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:31.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21889 '$2 == pid {print; exit 1}' 2026-02-22T20:08:31.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 21889 0.1 0.0 807920 52192 ? Ssl 19:53 0:01 rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:08:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:32.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:32.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21889 '$2 == pid {print; exit 1}' 2026-02-22T20:08:32.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:08:32.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:08:32.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=21889 '$2 == pid {print; exit 1}' 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:08:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:08:32.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:08:32.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:08:32.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:08:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:08:32.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:3 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:08:32.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:08:32.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:08:32.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:08:32.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:08:32.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_demote_image 100 2026-02-22T20:08:32.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:32.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:32.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:32.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:08:32.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:08:32.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:32.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:08:32.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_demote_image --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:08:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:08:32.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:08:32.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.21 bytes/sec: 24 MiB/s 2026-02-22T20:08:32.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test_demote_image 2026-02-22T20:08:32.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:32.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:32.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:32.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test_demote_image 2026-02-22T20:08:34.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:08:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirrors cluster1 2026-02-22T20:08:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:34.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:08:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:0 2026-02-22T20:08:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:08:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:08:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:08:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T20:08:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:08:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T20:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T20:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:34.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:08:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:1 2026-02-22T20:08:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:08:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:08:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:08:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T20:08:34.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:08:34.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:08:34.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:08:34.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T20:08:34.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:08:34.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:08:34.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:34.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T20:08:34.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:08:34.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:34.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:08:34.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:34.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:2 2026-02-22T20:08:34.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:08:34.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:08:34.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:34.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:08:34.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:34.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:3 2026-02-22T20:08:34.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:08:34.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:08:34.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:08:34.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T20:08:34.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:08:34.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:08:34.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:08:34.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T20:08:34.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:08:34.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:08:34.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:34.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T20:08:34.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:08:34.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:34.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:08:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test_demote_image 2026-02-22T20:08:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:08:34.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:34.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:34.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:34.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_demote_image.status 2026-02-22T20:08:34.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_demote_image.status 2026-02-22T20:08:34.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:08:34.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:08:34.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_demote_image.status 2026-02-22T20:08:34.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_demote_image.status 2026-02-22T20:08:34.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:08:34.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:34.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:08:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:08:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:08:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:34.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:08:34.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=175 2026-02-22T20:08:34.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:08:34.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:08:34.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:08:34.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:34.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:34.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:34.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:34.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:34.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:08:34.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:34.506 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:08:34.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=174 2026-02-22T20:08:34.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 175 = 174 2026-02-22T20:08:34.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:08:34.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:08:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:34.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:08:34.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:34.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:08:34.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=174 2026-02-22T20:08:34.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 175 = 174 2026-02-22T20:08:34.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:08:34.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.8 2026-02-22T20:08:35.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:35.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:35.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:35.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:35.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:35.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:08:35.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:35.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:08:35.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=174 2026-02-22T20:08:35.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 175 = 174 2026-02-22T20:08:35.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:08:35.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1.6 2026-02-22T20:08:37.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:37.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:37.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:37.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:37.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:37.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:08:37.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:08:37.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=174 2026-02-22T20:08:37.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 175 = 174 2026-02-22T20:08:37.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:08:37.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:08:39.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:39.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:39.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:39.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:39.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:08:39.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:08:39.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:39.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:08:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=175 2026-02-22T20:08:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 175 = 175 2026-02-22T20:08:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_demote_image up+unknown 2026-02-22T20:08:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:39.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:08:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:39.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_demote_image up+unknown '' '' 2026-02-22T20:08:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:08:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:40.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_demote_image 2026-02-22T20:08:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:08:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 22f785c3-8fce-4561-b0d3-a0dffd391329 2026-02-22T20:08:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:08:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:08:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:08:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:37 2026-02-22T20:08:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:08:40.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:08:40.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:08:40.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:08:40.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:39' 2026-02-22T20:08:40.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:08:40.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:08:40.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:08:40.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:08:40.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:08:40.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:08:40.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:08:40.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:08:40.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:08:40.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:08:40.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:08:40.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:37' 2026-02-22T20:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_demote_image up+unknown '2026-02-22 20:08:37' 2026-02-22T20:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:08:37' 2026-02-22T20:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:40.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:08:40.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING1144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:08:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}2026-02-22 20:08:38test_demote_image22f785c3-8fce-4561-b0d3-a0dffd391329up+unknownremote image is not primary44564466mirror.1trial0172026-02-22 20:08:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 20:08:39' 2026-02-22T20:08:40.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:08:40.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:08:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:37' 2026-02-22T20:08:40.538 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:08:40.540 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:08:40.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:08:40.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:08:40.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:08:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_demote_image up+unknown 2026-02-22T20:08:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:08:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:40.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:41.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+unknown '' '' 2026-02-22T20:08:41.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:41.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:41.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:08:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:41.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-22T20:08:41.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:08:41.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 22f785c3-8fce-4561-b0d3-a0dffd391329 2026-02-22T20:08:41.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:08:41.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:08:41.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:08:41.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:39 2026-02-22T20:08:41.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:08:41.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:08:41.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:08:41.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:08:41.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:37' 2026-02-22T20:08:41.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:08:41.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:08:41.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:08:41.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image demoted 2026-02-22T20:08:41.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:08:41.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:08:41.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:08:41.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:08:41.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:08:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:08:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:08:41.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:08:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:39' 2026-02-22T20:08:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_demote_image up+unknown '2026-02-22 20:08:39' 2026-02-22T20:08:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:08:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:08:39' 2026-02-22T20:08:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:08:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING1156435653mirror.0trial01719.2.3-689-gf90f4eaetrueOK56475657mirror.1trial01719.2.3-689-gf90f4eaefalseOK56515662mirror.2trial01719.2.3-689-gf90f4eaefalseOK56595665mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}56475657mirror.1trial0172026-02-22 20:08:38cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:08:37test_demote_image22f785c3-8fce-4561-b0d3-a0dffd391329up+unknownremote image demoted56435653mirror.0trial0172026-02-22 20:08:39cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:08:37' 2026-02-22T20:08:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:08:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:08:41.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:39' 2026-02-22T20:08:41.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:08:41.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:08:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:08:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:08:41.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:08:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test_demote_image 2026-02-22T20:08:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:08:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_demote_image.export 2026-02-22T20:08:41.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_demote_image.export 2026-02-22T20:08:41.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:08:41.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.export 2026-02-22T20:08:41.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_demote_image.export 2026-02-22T20:08:41.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_demote_image.export 2026-02-22T20:08:41.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:08:41.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.export 2026-02-22T20:08:41.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.export 2026-02-22T20:08:41.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test_demote_image /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.export 2026-02-22T20:08:41.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:08:41.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.export 2026-02-22T20:08:41.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:08:41.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.export 2026-02-22T20:08:41.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.export 2026-02-22T20:08:42.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:42.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test_demote_image 2026-02-22T20:08:42.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:42.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:42.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:42.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:08:42.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_demote_image 2026-02-22T20:08:42.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:08:42.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test_demote_image 2026-02-22T20:08:42.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:42.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:42.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:42.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_demote_image started 2026-02-22T20:08:42.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:42.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:42.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:42.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:08:42.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:08:42.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:42.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:43.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_demote_image started 2026-02-22T20:08:43.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:43.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:43.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:43.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:08:43.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:08:43.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:08:43.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:43.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:08:43.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:08:43.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:43.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:08:43.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:08:43.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:08:43.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:43.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:08:43.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:08:43.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:08:43.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:43.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:43.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:08:43.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:43.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:43.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:08:43.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:08:43.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:08:43.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:08:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:43.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:08:43.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-22T20:08:43.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:43.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_demote_image 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:08:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:08:43.697 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:08:43.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:43.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:43.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:43.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:08:43.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:08:43.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:08:43.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:08:43.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:08:43.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:08:43.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:08:43.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:08:43.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:43.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_demote_image up+stopped 2026-02-22T20:08:43.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:43.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:43.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:43.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:08:43.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:43.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:43.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:43.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:44.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+stopped '' '' 2026-02-22T20:08:44.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:44.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:44.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:44.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:08:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:44.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-22T20:08:44.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:08:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 22f785c3-8fce-4561-b0d3-a0dffd391329 2026-02-22T20:08:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:08:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:08:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:08:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:44 2026-02-22T20:08:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:08:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:08:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:08:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790922,"replay_state":"idle"} 2026-02-22T20:08:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:43 2026-02-22T20:08:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:08:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 181 .mirror.primary.22f785c3-8fce-4561-b0d3-a0dffd391329.c47fff30-d77c-4256-b590-3ab909fcdd70 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:08:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:08:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:08:44.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:08:44.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:08:44.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:08:44.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:08:44.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:08:44.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:08:44.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:08:44.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:08:44.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:08:44.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:08:44.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:44' 2026-02-22T20:08:44.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_demote_image up+stopped '2026-02-22 20:08:44' 2026-02-22T20:08:44.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:44.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:08:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:08:44' 2026-02-22T20:08:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:08:44.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK256435653mirror.0trial01719.2.3-689-gf90f4eaetrueOK56475657mirror.1trial01719.2.3-689-gf90f4eaefalseOK56515662mirror.2trial01719.2.3-689-gf90f4eaefalseOK56595665mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}56475657mirror.1trial0172026-02-22 20:08:38cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:08:42test_demote_image22f785c3-8fce-4561-b0d3-a0dffd391329up+stoppedlocal image is primary56435653mirror.0trial0172026-02-22 20:08:44cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790922,"replay_state":"idle"}2026-02-22 20:08:43' 2026-02-22T20:08:44.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:08:44.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:08:44.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:44' 2026-02-22T20:08:44.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:08:44.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:08:44.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:08:44.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:08:44.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:08:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:44.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_demote_image up+replaying 2026-02-22T20:08:44.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:44.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:44.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:44.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:08:44.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:44.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:44.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:44.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_demote_image up+replaying '' '' 2026-02-22T20:08:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:08:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_demote_image 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 22f785c3-8fce-4561-b0d3-a0dffd391329 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790922,"replay_state":"idle"} 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:43 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:44' 2026-02-22T20:08:45.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:08:45.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:08:45.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:08:45.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790922,"replay_state":"idle"} 2026-02-22T20:08:45.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:08:45.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:08:45.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:08:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:08:45.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:08:45.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:08:45.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:08:45.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:08:45.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:43' 2026-02-22T20:08:45.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_demote_image up+replaying '2026-02-22 20:08:43' 2026-02-22T20:08:45.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:45.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:45.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:45.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:08:45.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:08:43' 2026-02-22T20:08:45.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:45.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:45.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:08:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK244524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:08:42cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}2026-02-22 20:08:38test_demote_image22f785c3-8fce-4561-b0d3-a0dffd391329up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790922,"replay_state":"idle"}44564466mirror.1trial0172026-02-22 20:08:43cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:08:44' 2026-02-22T20:08:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:08:45.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:08:45.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:43' 2026-02-22T20:08:45.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:08:45.956 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:08:45.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:08:45.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:08:45.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:08:45.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:45.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:45.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster1 mirror test_demote_image 2026-02-22T20:08:45.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:45.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:45.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:45.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:08:45.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:08:45.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster1 mirror test_demote_image 2026-02-22T20:08:45.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:45.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:45.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:08:45.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_demote_image 2026-02-22T20:08:45.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 rm mirror/test_demote_image 2026-02-22T20:08:47.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:08:47.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:47.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:08:47.664 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: request image resync when remote is not primary' 2026-02-22T20:08:47.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:08:47.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:08:47.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:08:47 TEST: request image resync when remote is not primary 2026-02-22T20:08:47.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:08:47 TEST: request image resync when remote is not primary 2026-02-22T20:08:47.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_resync_image=test_resync_image 2026-02-22T20:08:47.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test_resync_image snapshot 2026-02-22T20:08:47.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:47.667 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test_resync_image 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:47.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:47.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:47.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:08:47.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:08:47.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:08:47.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test_resync_image 2026-02-22T20:08:47.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:08:47.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:47.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:08:47.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:47.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test_resync_image snapshot 2026-02-22T20:08:47.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:47.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:47.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:47.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:08:47.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test_resync_image snapshot 2026-02-22T20:08:48.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:08:48.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test_resync_image 2026-02-22T20:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_resync_image': 2026-02-22T20:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 170ac8680691 2026-02-22T20:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.170ac8680691 2026-02-22T20:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:08:48.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:08:48.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:08:47 2026 2026-02-22T20:08:48.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:08:47 2026 2026-02-22T20:08:48.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:08:47 2026 2026-02-22T20:08:48.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 170ac8680691 2026-02-22T20:08:48.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:08:48.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:08:48.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 009b5ea4-d648-4813-9dab-863895fb6b3e 2026-02-22T20:08:48.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:08:48.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_resync_image 100 2026-02-22T20:08:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:08:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:08:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:08:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_resync_image --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:08:48.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:08:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:08:48.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.21 bytes/sec: 24 MiB/s 2026-02-22T20:08:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_resync_image 2026-02-22T20:08:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_resync_image stopped 2026-02-22T20:08:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:48.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:08:48.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:08:48.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:48.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:49.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_resync_image stopped 2026-02-22T20:08:49.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:49.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:49.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:49.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:08:49.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:08:49.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:08:49.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_resync_image 2026-02-22T20:08:49.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:08:49.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:08:49.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:49.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:08:49.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:08:49.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:08:49.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:49.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:08:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:08:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:08:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_resync_image 2026-02-22T20:08:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:49.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:08:49.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:49.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:49.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:08:49.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:08:49.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:08:49.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:49.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:08:49.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:08:49.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:49.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:49.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_resync_image 2026-02-22T20:08:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:08:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:08:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:08:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:08:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:08:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:08:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:08:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:08:49.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:49.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:08:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:08:49.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:49.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:49.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:49.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:2 rbd mirror status mirror/test_resync_image 2026-02-22T20:08:49.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:2 2026-02-22T20:08:49.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-22T20:08:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:08:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:08:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:08:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:49.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:2 cluster2 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:2 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:2 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:08:49.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-22T20:08:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:08:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:08:49.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-22T20:08:49.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:08:49.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:08:49.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:08:49.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:08:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:08:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:49.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:3 rbd mirror status mirror/test_resync_image 2026-02-22T20:08:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:3 2026-02-22T20:08:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-22T20:08:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:3 cluster2 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:3 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:49.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-22T20:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:3 2026-02-22T20:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-22T20:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-22T20:08:49.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:08:49.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:08:49.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:08:49.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:08:49.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:08:49.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:49.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:49.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:49.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T20:08:50.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:08:50.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:08:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_resync_image 2026-02-22T20:08:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:08:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:08:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:08:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:50.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:08:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:08:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:08:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:08:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:08:50.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:08:50.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:08:50.554 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:08:50.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:50.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:50.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:08:50.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:08:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:08:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:08:50.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:08:50.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:50.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_resync_image 2026-02-22T20:08:50.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:08:50.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:08:50.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:08:50.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:08:50.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:08:50.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:08:50.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:08:50.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:50.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:50.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:08:50.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:08:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:08:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:08:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:08:50.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_resync_image 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_resync_image started 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:50.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_resync_image started 2026-02-22T20:08:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:08:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:08:51.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:08:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_resync_image 2026-02-22T20:08:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:08:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:08:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:08:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:08:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:51.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:08:51.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:08:51.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:08:51.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:08:51.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_resync_image 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:08:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:08:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:08:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:08:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:08:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:08:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:08:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:08:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:08:51.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:08:51.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:08:51.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:08:51.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:08:51.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:08:51.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:08:51.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:08:51.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:08:51.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:08:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:08:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:08:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_resync_image 2026-02-22T20:08:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:08:51.790 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:08:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:08:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test_resync_image 2026-02-22T20:08:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:51.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test_resync_image 2026-02-22T20:08:51.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 178 2026-02-22T20:08:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test_resync_image 2026-02-22T20:08:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:08:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_resync_image.status 2026-02-22T20:08:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_resync_image.status 2026-02-22T20:08:51.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:08:51.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.status 2026-02-22T20:08:51.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_resync_image.status 2026-02-22T20:08:51.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_resync_image.status 2026-02-22T20:08:51.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:08:51.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.status 2026-02-22T20:08:51.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test_resync_image /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.status 2026-02-22T20:08:51.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.status 2026-02-22T20:08:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test_resync_image --format xml 2026-02-22T20:08:51.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:51.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:08:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=178 2026-02-22T20:08:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:08:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:08:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:08:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_resync_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.status 2026-02-22T20:08:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.status 2026-02-22T20:08:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_resync_image --format xml 2026-02-22T20:08:52.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:52.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:08:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=177 2026-02-22T20:08:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 178 = 177 2026-02-22T20:08:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:08:52.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:08:52.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_resync_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.status 2026-02-22T20:08:52.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:52.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:52.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:52.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.status 2026-02-22T20:08:52.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_resync_image --format xml 2026-02-22T20:08:52.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:08:52.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:08:52.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=178 2026-02-22T20:08:52.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 178 = 178 2026-02-22T20:08:52.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:52.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_resync_image 2026-02-22T20:08:52.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:52.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:52.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:52.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:08:52.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_resync_image up+replaying 2026-02-22T20:08:52.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:52.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:52.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:52.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:08:52.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:52.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:52.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:52.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:53.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_resync_image up+replaying '' '' 2026-02-22T20:08:53.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:53.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:53.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:53.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:08:53.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:53.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:53.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:53.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:53.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_resync_image 2026-02-22T20:08:53.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:08:53.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 009b5ea4-d648-4813-9dab-863895fb6b3e 2026-02-22T20:08:53.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:08:53.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790928,"replay_state":"idle"} 2026-02-22T20:08:53.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:08:53.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:50 2026-02-22T20:08:53.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:08:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:08:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:08:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:08:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:52' 2026-02-22T20:08:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:08:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:08:53.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:08:53.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790928,"replay_state":"idle"} 2026-02-22T20:08:53.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:08:53.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:08:53.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:08:53.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:08:53.709 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:08:53.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:08:53.710 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:08:53.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:08:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:50' 2026-02-22T20:08:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_resync_image up+replaying '2026-02-22 20:08:50' 2026-02-22T20:08:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:53.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:08:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:08:50' 2026-02-22T20:08:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:53.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:08:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK256435653mirror.0trial01719.2.3-689-gf90f4eaetrueOK56475657mirror.1trial01719.2.3-689-gf90f4eaefalseOK56515662mirror.2trial01719.2.3-689-gf90f4eaefalseOK56595665mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}56475657mirror.1trial0172026-02-22 20:08:38cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:08:52test_resync_image009b5ea4-d648-4813-9dab-863895fb6b3eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790928,"replay_state":"idle"}56435653mirror.0trial0172026-02-22 20:08:50cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:08:52' 2026-02-22T20:08:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:08:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:08:53.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:50' 2026-02-22T20:08:53.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:08:53.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:08:53.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:08:53.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:08:53.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:08:53.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:53.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:53.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_resync_image up+stopped 2026-02-22T20:08:53.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:53.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:53.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:53.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:08:53.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:53.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:53.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:53.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:54.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_resync_image up+stopped '' '' 2026-02-22T20:08:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:08:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:08:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_resync_image 2026-02-22T20:08:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:08:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 009b5ea4-d648-4813-9dab-863895fb6b3e 2026-02-22T20:08:54.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:52 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790928,"replay_state":"idle"} 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:50 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 177 .mirror.primary.009b5ea4-d648-4813-9dab-863895fb6b3e.b804c40c-aa99-40fc-a54c-dd4e28630017 (peer_uuids:[]) 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 178 .mirror.primary.009b5ea4-d648-4813-9dab-863895fb6b3e.3d33fd9c-2e44-49f9-8143-d9919aa49056 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:08:54.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:08:54.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:08:54.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:08:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:08:54.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:08:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:08:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:08:54.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:08:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:08:54.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:08:54.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:08:54.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:08:54.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:52' 2026-02-22T20:08:54.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_resync_image up+stopped '2026-02-22 20:08:52' 2026-02-22T20:08:54.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:54.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:54.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:54.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:08:54.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:08:52' 2026-02-22T20:08:54.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:54.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:54.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:08:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK244524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:08:52cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}2026-02-22 20:08:38test_resync_image009b5ea4-d648-4813-9dab-863895fb6b3eup+stoppedlocal image is primary44564466mirror.1trial0172026-02-22 20:08:52cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790928,"replay_state":"idle"}2026-02-22 20:08:50' 2026-02-22T20:08:54.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:08:54.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:08:54.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:52' 2026-02-22T20:08:54.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:08:54.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:08:54.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:08:54.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:08:54.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:08:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_resync_image 100 2026-02-22T20:08:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:54.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:08:54.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:08:54.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:08:54.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:08:54.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_resync_image --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:08:54.887 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:08:54.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:08:54.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.2 bytes/sec: 24 MiB/s 2026-02-22T20:08:54.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test_resync_image 2026-02-22T20:08:54.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:54.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:54.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:54.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test_resync_image 2026-02-22T20:08:56.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:08:56.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ request_resync_image cluster1 mirror test_resync_image test_resync_image_id 2026-02-22T20:08:56.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:56.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:56.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id_var_name=test_resync_image_id 2026-02-22T20:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test_resync_image 2026-02-22T20:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_resync_image 2026-02-22T20:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test_resync_image 2026-02-22T20:08:56.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:08:56.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'test_resync_image_id='\''1615f30b5b2'\''' 2026-02-22T20:08:56.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test_resync_image_id=1615f30b5b2 2026-02-22T20:08:56.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'test -n "$test_resync_image_id"' 2026-02-22T20:08:56.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -n 1615f30b5b2 2026-02-22T20:08:56.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test_resync_image 2026-02-22T20:08:56.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Flagged image for resync from primary 2026-02-22T20:08:56.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_resync_image up+unknown 'remote image is not primary' 2026-02-22T20:08:56.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:08:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:08:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:08:57.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_resync_image up+unknown 'remote image is not primary' '' 2026-02-22T20:08:57.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:57.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:57.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:57.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:08:57.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:08:57.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:57.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:57.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:57.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_resync_image 2026-02-22T20:08:57.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:08:57.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 009b5ea4-d648-4813-9dab-863895fb6b3e 2026-02-22T20:08:57.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:08:57.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:08:57.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:08:57.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:52 2026-02-22T20:08:57.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:08:57.266 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:08:57.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:08:57.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:08:57.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:56' 2026-02-22T20:08:57.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:08:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:08:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:57.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:08:59.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_resync_image up+unknown 'remote image is not primary' '' 2026-02-22T20:08:59.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:59.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:59.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:59.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:08:59.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:08:59.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:59.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:59.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:59.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_resync_image 2026-02-22T20:08:59.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:08:59.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 009b5ea4-d648-4813-9dab-863895fb6b3e 2026-02-22T20:08:59.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:08:59.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:08:59.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:08:59.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:57 2026-02-22T20:08:59.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:08:59.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:08:59.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:08:59.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:08:59.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:59' 2026-02-22T20:08:59.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:08:59.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:08:59.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*remote image is not primary' 2026-02-22T20:08:59.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:08:59.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:08:59.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:08:59.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:08:59.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:08:59.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:08:59.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:08:59.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:08:59.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:08:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:57' 2026-02-22T20:08:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_resync_image up+unknown '2026-02-22 20:08:57' 2026-02-22T20:08:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:08:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:59.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:08:59.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:08:57' 2026-02-22T20:08:59.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:08:59.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:08:59.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:08:59.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING1144524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:08:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}2026-02-22 20:08:38test_resync_image009b5ea4-d648-4813-9dab-863895fb6b3eup+unknownremote image is not primary44564466mirror.1trial0172026-02-22 20:08:57cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image is not primary2026-02-22 20:08:59' 2026-02-22T20:08:59.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:08:59.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:08:59.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:57' 2026-02-22T20:08:59.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:08:59.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:08:59.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:08:59.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:08:59.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:08:59.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:59.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:08:59.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_resync_image up+unknown 'remote image is not primary' 2026-02-22T20:08:59.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:08:59.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:08:59.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:08:59.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:08:59.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:08:59.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:08:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:08:59.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:00.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_resync_image up+unknown 'remote image is not primary' '' 2026-02-22T20:09:00.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:00.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:00.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:00.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:09:00.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:09:00.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:00.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:00.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:00.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_resync_image 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 009b5ea4-d648-4813-9dab-863895fb6b3e 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:59 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:08:57' 2026-02-22T20:09:00.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:09:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:09:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*remote image is not primary' 2026-02-22T20:09:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:09:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:09:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:09:00.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:09:00.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:09:00.391 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:09:00.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:09:00.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:09:00.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:09:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:59' 2026-02-22T20:09:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_resync_image up+unknown '2026-02-22 20:08:59' 2026-02-22T20:09:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:09:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:08:59' 2026-02-22T20:09:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:00.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:09:00.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING1156435653mirror.0trial01719.2.3-689-gf90f4eaetrueOK56475657mirror.1trial01719.2.3-689-gf90f4eaefalseOK56515662mirror.2trial01719.2.3-689-gf90f4eaefalseOK56595665mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}56475657mirror.1trial0172026-02-22 20:08:38cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:08:57test_resync_image009b5ea4-d648-4813-9dab-863895fb6b3eup+unknownremote image is not primary56435653mirror.0trial0172026-02-22 20:08:59cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:08:57' 2026-02-22T20:09:00.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:09:00.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:09:00.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:08:59' 2026-02-22T20:09:00.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:09:00.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:09:00.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:09:00.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:09:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:09:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test_resync_image 2026-02-22T20:09:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:09:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_resync_image 2026-02-22T20:09:00.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:09:00.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test_resync_image 2026-02-22T20:09:00.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:00.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_resync_image started 2026-02-22T20:09:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:09:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:09:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:00.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:01.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_resync_image started 2026-02-22T20:09:01.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:01.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:01.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:01.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:09:01.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:09:01.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:09:01.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:09:01.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:01.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:01.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:09:01.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:09:01.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:09:01.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:01.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:01.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:01.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:01.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:09:01.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:09:01.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:09:01.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:01.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:09:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:09:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:09:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:01.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:09:01.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:09:01.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:09:01.796 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:09:01.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T20:09:01.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:01.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:09:03.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_resync_image started 2026-02-22T20:09:03.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:03.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:03.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:03.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:09:03.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:09:03.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:09:03.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:03.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:09:03.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:09:03.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:03.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:03.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:09:03.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:09:03.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:03.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:09:03.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:09:03.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:09:03.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:03.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:03.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:09:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:09:03.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:09:03.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:03.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:03.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:03.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:03.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:09:03.866 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:09:03.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:09:03.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:03.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:03.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:09:03.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:03.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:03.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:09:03.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:09:03.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:03.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:03.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:03.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:03.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:03.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:03.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:03.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:09:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:09:03.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:09:03.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:03.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:03.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:03.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:03.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:09:03.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:09:03.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:09:03.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:09:03.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:09:03.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:09:03.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:09:03.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:03.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_resync_image up+stopped 2026-02-22T20:09:03.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:03.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:03.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:03.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:09:03.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:03.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:03.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:03.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_resync_image up+stopped '' '' 2026-02-22T20:09:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:04.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:09:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:04.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_resync_image 2026-02-22T20:09:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 009b5ea4-d648-4813-9dab-863895fb6b3e 2026-02-22T20:09:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:09:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:09:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:09:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:04 2026-02-22T20:09:04.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:09:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:09:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790940,"replay_state":"idle"} 2026-02-22T20:09:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:03 2026-02-22T20:09:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:09:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 185 .mirror.primary.009b5ea4-d648-4813-9dab-863895fb6b3e.e2a9a96a-383a-4b28-af32-f0a718f1a722 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:09:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:09:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:09:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:09:04.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:09:04.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:09:04.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:09:04.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:09:04.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:09:04.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:09:04.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:09:04.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:09:04.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:09:04.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:04' 2026-02-22T20:09:04.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_resync_image up+stopped '2026-02-22 20:09:04' 2026-02-22T20:09:04.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:04.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:04.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:04.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:09:04.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:09:04' 2026-02-22T20:09:04.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:04.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:04.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:09:05.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK256435653mirror.0trial01719.2.3-689-gf90f4eaetrueOK56475657mirror.1trial01719.2.3-689-gf90f4eaefalseOK56515662mirror.2trial01719.2.3-689-gf90f4eaefalseOK56595665mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}56475657mirror.1trial0172026-02-22 20:09:04cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:09:02test_resync_image009b5ea4-d648-4813-9dab-863895fb6b3eup+stoppedlocal image is primary56435653mirror.0trial0172026-02-22 20:09:04cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790940,"replay_state":"idle"}2026-02-22 20:09:03' 2026-02-22T20:09:05.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:09:05.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:09:05.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:04' 2026-02-22T20:09:05.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:09:05.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:09:05.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:09:05.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:09:05.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:09:05.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:05.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:05.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_resync_image up+replaying 2026-02-22T20:09:05.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:05.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:05.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:05.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:09:05.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:05.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:05.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:05.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_resync_image up+replaying '' '' 2026-02-22T20:09:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:09:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_resync_image 2026-02-22T20:09:06.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:06.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 009b5ea4-d648-4813-9dab-863895fb6b3e 2026-02-22T20:09:06.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:09:06.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790940,"replay_state":"idle"} 2026-02-22T20:09:06.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:09:06.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:03 2026-02-22T20:09:06.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:06.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:09:06.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:09:06.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:09:06.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:04' 2026-02-22T20:09:06.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:09:06.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:09:06.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:09:06.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790940,"replay_state":"idle"} 2026-02-22T20:09:06.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:09:06.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:09:06.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:09:06.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:09:06.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:09:06.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:09:06.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:09:06.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:09:06.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:03' 2026-02-22T20:09:06.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_resync_image up+replaying '2026-02-22 20:09:03' 2026-02-22T20:09:06.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:06.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:06.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:06.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:09:06.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:09:03' 2026-02-22T20:09:06.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:06.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:06.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:09:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK244524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:09:02cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}2026-02-22 20:09:04test_resync_image009b5ea4-d648-4813-9dab-863895fb6b3eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790940,"replay_state":"idle"}44564466mirror.1trial0172026-02-22 20:09:03cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:09:04' 2026-02-22T20:09:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:09:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:09:06.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:03' 2026-02-22T20:09:06.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:09:06.122 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:09:06.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:09:06.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:09:06.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:09:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test_resync_image 2026-02-22T20:09:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:06.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:09:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_resync_image.export 2026-02-22T20:09:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_resync_image.export 2026-02-22T20:09:06.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:09:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.export 2026-02-22T20:09:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_resync_image.export 2026-02-22T20:09:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_resync_image.export 2026-02-22T20:09:06.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:09:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.export 2026-02-22T20:09:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.export 2026-02-22T20:09:06.129 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test_resync_image /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.export 2026-02-22T20:09:06.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:09:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test_resync_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.export 2026-02-22T20:09:06.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:09:06.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.export 2026-02-22T20:09:06.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.export 2026-02-22T20:09:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster1 mirror test_resync_image 2026-02-22T20:09:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:06.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster1 mirror test_resync_image 2026-02-22T20:09:06.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:06.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:06.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:06.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_resync_image 2026-02-22T20:09:06.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 rm mirror/test_resync_image 2026-02-22T20:09:07.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:09:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z '' ']' 2026-02-22T20:09:07.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: request image resync when remote is not primary and daemon is offline' 2026-02-22T20:09:07.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:09:07.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:09:07.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:09:07 TEST: request image resync when remote is not primary and daemon is offline 2026-02-22T20:09:07.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:09:07 TEST: request image resync when remote is not primary and daemon is offline 2026-02-22T20:09:07.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_resync_image=test_resync_image 2026-02-22T20:09:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test_resync_image snapshot 2026-02-22T20:09:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:09:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:09:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:09:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:09:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:09:07.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:09:07.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test_resync_image 2026-02-22T20:09:07.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:07.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:09:07.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:07.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:09:07.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:07.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:09:07.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:09:07.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:09:07.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test_resync_image 2026-02-22T20:09:07.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:09:07.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:07.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:09:07.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:07.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test_resync_image snapshot 2026-02-22T20:09:07.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:07.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:07.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:07.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:09:07.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test_resync_image snapshot 2026-02-22T20:09:08.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:09:08.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test_resync_image 2026-02-22T20:09:08.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_resync_image': 2026-02-22T20:09:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:09:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:09:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:09:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 173a914671cf 2026-02-22T20:09:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.173a914671cf 2026-02-22T20:09:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:09:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:09:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:09:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:09:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:09:07 2026 2026-02-22T20:09:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:09:07 2026 2026-02-22T20:09:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:09:07 2026 2026-02-22T20:09:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 173a914671cf 2026-02-22T20:09:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:09:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:09:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: c596c720-b4c1-44da-81d8-6d8beb6fbceb 2026-02-22T20:09:08.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:09:08.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_resync_image 100 2026-02-22T20:09:08.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:08.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:08.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:08.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:09:08.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:09:08.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:08.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:09:08.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_resync_image --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:09:08.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:09:08.269 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.2 bytes/sec: 24 MiB/s 2026-02-22T20:09:08.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_resync_image 2026-02-22T20:09:08.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:08.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:08.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:08.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_resync_image stopped 2026-02-22T20:09:08.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:08.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:08.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:08.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:09:08.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:09:08.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:08.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:09.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_resync_image stopped 2026-02-22T20:09:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:09:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:09:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:09:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:09:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:09:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:09.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:09.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:09:09.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:09:09.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:09.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:09:09.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:09:09.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:09:09.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:09:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:09.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:09:09.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:09:09.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:09.346 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:09.347 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:09:09.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:09:09.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:09:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:09.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:09.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:09.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:09:09.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:09.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:2 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:2 2026-02-22T20:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-22T20:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:09:09.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:2 cluster2 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:2 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:2 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-22T20:09:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:09:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:09:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-22T20:09:09.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:09:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:09:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:09:09.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:09.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:09:09.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:09:09.485 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:3 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:3 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:09.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:3 cluster2 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:3 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:3 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-22T20:09:09.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:09:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:09:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-22T20:09:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:09:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:09:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:09:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:09.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:09:09.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:09:09.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:09:09.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:09.556 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T20:09:10.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:09:10.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:09:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:09:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:09:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:09:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:10.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:09:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:09:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:09:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:09:10.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:10.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:10.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:10.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:10.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:09:10.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:09:10.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:09:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:09:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:09:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:09:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:09:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:09:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:10.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:09:10.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:09:10.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:10.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:10.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:10.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:10.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:10.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:10.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:10.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:09:10.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:09:10.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:09:10.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:10.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:10.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:10.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:09:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:09:10.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:09:10.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:09:10.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:09:10.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:10.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_resync_image 2026-02-22T20:09:10.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:10.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:10.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:10.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_resync_image started 2026-02-22T20:09:10.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:10.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:10.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:10.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:09:10.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:09:10.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:10.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_resync_image started 2026-02-22T20:09:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:09:11.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:09:11.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:09:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:09:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:09:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:11.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:09:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:09:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:09:11.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:09:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:09:11.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:09:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:09:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:09:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:11.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:11.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:09:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:09:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:09:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:09:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:09:11.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:09:11.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:09:11.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:09:11.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:11.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:09:11.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:09:11.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:09:11.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:09:11.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:09:11.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:09:11.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:09:11.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:11.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_resync_image 2026-02-22T20:09:11.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:09:11.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:11.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:11.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:11.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:09:11.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:09:11.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test_resync_image 2026-02-22T20:09:11.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:11.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:11.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:11.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test_resync_image 2026-02-22T20:09:12.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 182 2026-02-22T20:09:12.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test_resync_image 2026-02-22T20:09:12.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:09:12.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:12.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:12.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:12.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_resync_image.status 2026-02-22T20:09:12.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_resync_image.status 2026-02-22T20:09:12.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:09:12.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.status 2026-02-22T20:09:12.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_resync_image.status 2026-02-22T20:09:12.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_resync_image.status 2026-02-22T20:09:12.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:09:12.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.status 2026-02-22T20:09:12.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test_resync_image /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.status 2026-02-22T20:09:12.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:12.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:12.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:12.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.status 2026-02-22T20:09:12.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test_resync_image --format xml 2026-02-22T20:09:12.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:09:12.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:09:12.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=182 2026-02-22T20:09:12.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:09:12.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:09:12.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_resync_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.status 2026-02-22T20:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.status 2026-02-22T20:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_resync_image --format xml 2026-02-22T20:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:09:12.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:09:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=181 2026-02-22T20:09:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 182 = 181 2026-02-22T20:09:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:09:12.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:09:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_resync_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.status 2026-02-22T20:09:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.status 2026-02-22T20:09:12.864 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_resync_image --format xml 2026-02-22T20:09:12.865 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:09:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=182 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 182 = 182 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_resync_image 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_resync_image up+replaying 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:12.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_resync_image up+replaying '' '' 2026-02-22T20:09:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:09:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:13.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:13.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:13.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:13.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_resync_image 2026-02-22T20:09:13.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:13.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: c596c720-b4c1-44da-81d8-6d8beb6fbceb 2026-02-22T20:09:13.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:09:13.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790948,"replay_state":"idle"} 2026-02-22T20:09:13.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:09:13.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:09 2026-02-22T20:09:13.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:13.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:09:13.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:09:13.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:09:13.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:12' 2026-02-22T20:09:13.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:09:13.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:09:13.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:09:13.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790948,"replay_state":"idle"} 2026-02-22T20:09:13.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:09:13.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:09:13.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:09:13.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:09:13.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:09:13.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:09:13.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:09:13.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:09:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:09' 2026-02-22T20:09:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_resync_image up+replaying '2026-02-22 20:09:09' 2026-02-22T20:09:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:09:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:09:09' 2026-02-22T20:09:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:13.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:09:13.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK256435653mirror.0trial01719.2.3-689-gf90f4eaetrueOK56475657mirror.1trial01719.2.3-689-gf90f4eaefalseOK56515662mirror.2trial01719.2.3-689-gf90f4eaefalseOK56595665mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}56475657mirror.1trial0172026-02-22 20:09:04cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:09:12test_resync_imagec596c720-b4c1-44da-81d8-6d8beb6fbcebup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790948,"replay_state":"idle"}56435653mirror.0trial0172026-02-22 20:09:09cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:09:12' 2026-02-22T20:09:13.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:09:13.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:09:13.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:09' 2026-02-22T20:09:13.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:09:13.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:09:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:09:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:09:13.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:09:13.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:13.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:13.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_resync_image up+stopped 2026-02-22T20:09:13.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:09:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:13.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:14.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_resync_image up+stopped '' '' 2026-02-22T20:09:14.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:14.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:14.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:14.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:09:14.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:14.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:14.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:14.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:14.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_resync_image 2026-02-22T20:09:15.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: c596c720-b4c1-44da-81d8-6d8beb6fbceb 2026-02-22T20:09:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:09:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:09:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:09:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:12 2026-02-22T20:09:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:09:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:09:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790948,"replay_state":"idle"} 2026-02-22T20:09:15.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:09 2026-02-22T20:09:15.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:09:15.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 181 .mirror.primary.c596c720-b4c1-44da-81d8-6d8beb6fbceb.cd15d59f-b7ce-4be9-b512-9993d5ad16e0 (peer_uuids:[]) 2026-02-22T20:09:15.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 182 .mirror.primary.c596c720-b4c1-44da-81d8-6d8beb6fbceb.f25c87e0-7885-4224-9027-736f47352895 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:09:15.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:09:15.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:09:15.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:09:15.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:09:15.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:09:15.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:09:15.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:09:15.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:09:15.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:09:15.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:09:15.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:09:15.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:12' 2026-02-22T20:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_resync_image up+stopped '2026-02-22 20:09:12' 2026-02-22T20:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:15.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:15.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:09:15.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:09:12' 2026-02-22T20:09:15.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:15.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:15.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:09:15.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK244524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:09:12cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}2026-02-22 20:09:04test_resync_imagec596c720-b4c1-44da-81d8-6d8beb6fbcebup+stoppedlocal image is primary44564466mirror.1trial0172026-02-22 20:09:12cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790948,"replay_state":"idle"}2026-02-22 20:09:09' 2026-02-22T20:09:15.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:09:15.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:09:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:12' 2026-02-22T20:09:15.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:09:15.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:09:15.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:09:15.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:09:15.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:09:15.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:15.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:15.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirrors cluster1 2026-02-22T20:09:15.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:15.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:15.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:09:15.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:15.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:0 '' 2026-02-22T20:09:15.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:09:15.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:15.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:15.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:0 2026-02-22T20:09:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:0 2026-02-22T20:09:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:09:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:09:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:09:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:09:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:09:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:09:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:09:15.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:09:15.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:09:15.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=48318 2026-02-22T20:09:15.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 48318 ']' 2026-02-22T20:09:15.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 48318 2026-02-22T20:09:15.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:15.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:15.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:15.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48318 '$2 == pid {print; exit 1}' 2026-02-22T20:09:15.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 48318 0.7 0.0 775132 45800 ? Ssl 20:08 0:00 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:15.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:15.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:16.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:16.111 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48318 '$2 == pid {print; exit 1}' 2026-02-22T20:09:16.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:16.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:16.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48318 '$2 == pid {print; exit 1}' 2026-02-22T20:09:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:09:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:09:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:09:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:09:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:16.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:16.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:16.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:16.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:16.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:16.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:16.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:16.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:09:16.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:09:16.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:09:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-22T20:09:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:09:16.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:16.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:16.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:09:16.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:09:16.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:09:16.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:0 2026-02-22T20:09:16.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:09:16.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:16.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:09:16.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:09:16.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:16.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:16.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:16.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:16.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:16.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:09:16.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:16.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:16.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:16.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:09:16.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:09:16.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:09:16.181 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:16.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:1 '' 2026-02-22T20:09:16.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:09:16.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:16.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:16.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:1 2026-02-22T20:09:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:1 2026-02-22T20:09:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:09:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:09:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:09:16.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:09:16.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:09:16.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:09:16.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:09:16.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:09:16.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:09:16.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:09:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=48346 2026-02-22T20:09:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 48346 ']' 2026-02-22T20:09:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 48346 2026-02-22T20:09:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:16.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:16.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:16.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48346 '$2 == pid {print; exit 1}' 2026-02-22T20:09:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 48346 0.3 0.0 775128 41020 ? Ssl 20:08 0:00 rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:16.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:16.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:17.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:17.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48346 '$2 == pid {print; exit 1}' 2026-02-22T20:09:17.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:17.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:17.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48346 '$2 == pid {print; exit 1}' 2026-02-22T20:09:17.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:09:17.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:17.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:17.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:09:17.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:09:17.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:09:17.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-22T20:09:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:09:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:09:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:09:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:09:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:17.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:17.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:17.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:17.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:09:17.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:09:17.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:09:17.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:1 2026-02-22T20:09:17.262 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:09:17.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:09:17.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:09:17.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:17.273 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:2 '' 2026-02-22T20:09:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:09:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:17.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:2 2026-02-22T20:09:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:2 2026-02-22T20:09:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:09:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:09:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:09:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:09:17.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:09:17.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:09:17.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:09:17.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:09:17.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:09:17.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:09:17.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=48374 2026-02-22T20:09:17.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 48374 ']' 2026-02-22T20:09:17.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 48374 2026-02-22T20:09:17.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:17.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:17.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:17.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48374 '$2 == pid {print; exit 1}' 2026-02-22T20:09:17.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 48374 0.2 0.0 701364 37508 ? Ssl 20:08 0:00 rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:17.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:17.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:18.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:18.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48374 '$2 == pid {print; exit 1}' 2026-02-22T20:09:18.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:18.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:18.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48374 '$2 == pid {print; exit 1}' 2026-02-22T20:09:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:09:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:09:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:09:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:09:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:18.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:09:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:09:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:09:18.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:09:18.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:09:18.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-22T20:09:18.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:09:18.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:18.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:18.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:09:18.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:09:18.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:09:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:2 2026-02-22T20:09:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:09:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:09:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:09:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:18.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:18.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:18.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:18.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:09:18.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:09:18.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:18.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:09:18.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:18.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:09:18.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:09:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:09:18.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:3 '' 2026-02-22T20:09:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:09:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:18.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:3 2026-02-22T20:09:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:3 2026-02-22T20:09:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:09:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:09:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:09:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:09:18.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:09:18.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:09:18.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:09:18.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:09:18.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:09:18.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:09:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=48448 2026-02-22T20:09:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 48448 ']' 2026-02-22T20:09:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 48448 2026-02-22T20:09:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:18.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:18.385 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48448 '$2 == pid {print; exit 1}' 2026-02-22T20:09:18.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 48448 0.2 0.0 701372 37568 ? Ssl 20:08 0:00 rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:18.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:18.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:19.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:19.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48448 '$2 == pid {print; exit 1}' 2026-02-22T20:09:19.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:19.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:19.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=48448 '$2 == pid {print; exit 1}' 2026-02-22T20:09:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:09:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:09:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:09:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:09:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:19.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:19.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:09:19.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:19.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:19.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:09:19.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:19.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:09:19.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:09:19.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:09:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-22T20:09:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:09:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:09:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:09:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:19.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:09:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:09:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:19.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:09:19.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:09:19.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:09:19.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:3 2026-02-22T20:09:19.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:09:19.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:19.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:09:19.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:09:19.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:19.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:19.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:19.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:19.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:09:19.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:09:19.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:19.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:09:19.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:19.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:09:19.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:09:19.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:09:19.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_resync_image 100 2026-02-22T20:09:19.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:19.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:19.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:19.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:09:19.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:09:19.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:19.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:09:19.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_resync_image --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:09:19.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:09:19.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:09:19.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.2 bytes/sec: 24 MiB/s 2026-02-22T20:09:19.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test_resync_image 2026-02-22T20:09:19.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:19.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:19.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:19.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test_resync_image 2026-02-22T20:09:21.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:09:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ request_resync_image cluster1 mirror test_resync_image test_resync_image_id 2026-02-22T20:09:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:21.168 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id_var_name=test_resync_image_id 2026-02-22T20:09:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test_resync_image 2026-02-22T20:09:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:09:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_resync_image 2026-02-22T20:09:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test_resync_image 2026-02-22T20:09:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:09:21.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'test_resync_image_id='\''16153f7333cc'\''' 2026-02-22T20:09:21.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test_resync_image_id=16153f7333cc 2026-02-22T20:09:21.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'test -n "$test_resync_image_id"' 2026-02-22T20:09:21.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -n 16153f7333cc 2026-02-22T20:09:21.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test_resync_image 2026-02-22T20:09:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Flagged image for resync from primary 2026-02-22T20:09:21.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirrors cluster1 2026-02-22T20:09:21.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:21.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:0 2026-02-22T20:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:21.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:21.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:1 2026-02-22T20:09:21.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:09:21.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:21.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:2 2026-02-22T20:09:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:09:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:09:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:09:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T20:09:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:09:21.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:09:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:09:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T20:09:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:09:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:09:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T20:09:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:21.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:3 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T20:09:21.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:21.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:21.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:21.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_resync_image up+unknown 'remote image is not primary' 2026-02-22T20:09:21.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:21.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:21.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:21.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:09:21.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:09:21.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:21.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:21.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_resync_image up+unknown 'remote image is not primary' '' 2026-02-22T20:09:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:09:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:09:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:22.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:22.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:22.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_resync_image 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: c596c720-b4c1-44da-81d8-6d8beb6fbceb 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:17 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:09:22.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:09:22.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:09:22.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:22.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:09:24.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_resync_image up+unknown 'remote image is not primary' '' 2026-02-22T20:09:24.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:24.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:24.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:24.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:09:24.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:09:24.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:24.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:24.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:24.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_resync_image 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: c596c720-b4c1-44da-81d8-6d8beb6fbceb 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:22 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:09:24.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:09:24.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:09:24.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*remote image is not primary' 2026-02-22T20:09:24.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:09:24.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:09:24.591 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:09:24.592 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:09:24.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:09:24.593 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:09:24.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:09:24.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:09:24.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:09:24.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:22' 2026-02-22T20:09:24.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_resync_image up+unknown '2026-02-22 20:09:22' 2026-02-22T20:09:24.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:24.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:24.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:24.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:09:24.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:09:22' 2026-02-22T20:09:24.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:24.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:24.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:09:24.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING244524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:09:22cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5down+unknownstatus not foundtest_resync_imagec596c720-b4c1-44da-81d8-6d8beb6fbcebup+unknownremote image is not primary44564466mirror.1trial0172026-02-22 20:09:22cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5down+unknownstatus not found' 2026-02-22T20:09:24.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:09:24.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:09:24.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:22' 2026-02-22T20:09:24.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:09:24.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:09:24.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:09:24.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:09:24.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:09:24.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:24.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:24.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_resync_image up+unknown 'remote image is not primary' 2026-02-22T20:09:24.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:24.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:24.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:24.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:09:24.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:09:24.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:24.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:24.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:25.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_resync_image up+unknown 'remote image is not primary' '' 2026-02-22T20:09:25.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:25.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:25.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:25.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:09:25.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:09:25.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:25.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:25.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:25.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_resync_image 2026-02-22T20:09:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: c596c720-b4c1-44da-81d8-6d8beb6fbceb 2026-02-22T20:09:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:09:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790952,"remote_snapshot_timestamp":1771790960,"replay_state":"idle"} 2026-02-22T20:09:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:25 2026-02-22T20:09:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:09:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:09:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:09:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:22' 2026-02-22T20:09:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:09:25.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:09:25.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:25.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:09:27.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_resync_image up+unknown 'remote image is not primary' '' 2026-02-22T20:09:27.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:27.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:27.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:27.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:09:27.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:09:27.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:27.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:27.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:27.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_resync_image 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: c596c720-b4c1-44da-81d8-6d8beb6fbceb 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.2 on trial017 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:26 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:22' 2026-02-22T20:09:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:09:27.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:09:27.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*remote image is not primary' 2026-02-22T20:09:27.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:09:27.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:27.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 4 2026-02-22T20:09:31.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_resync_image up+unknown 'remote image is not primary' '' 2026-02-22T20:09:31.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:31.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:31.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:31.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:09:31.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'description_pattern=remote image is not primary' 2026-02-22T20:09:31.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:31.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:31.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:31.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_resync_image 2026-02-22T20:09:31.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:31.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: c596c720-b4c1-44da-81d8-6d8beb6fbceb 2026-02-22T20:09:31.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:09:31.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:09:31.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.2 on trial017 2026-02-22T20:09:31.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:31 2026-02-22T20:09:31.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:31.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:09:31.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:09:31.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:09:31.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:27' 2026-02-22T20:09:31.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:09:31.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:09:31.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*remote image is not primary' 2026-02-22T20:09:31.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:09:31.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:09:31.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:09:31.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:09:31.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:09:31.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:09:31.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.2 on trial017 2026-02-22T20:09:31.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:09:31.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:09:31.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:31' 2026-02-22T20:09:31.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_resync_image up+unknown '2026-02-22 20:09:31' 2026-02-22T20:09:31.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:31.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:31.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:31.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:09:31.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:09:31' 2026-02-22T20:09:31.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:31.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:31.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:09:31.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING1157715776mirror.0trial01719.2.3-689-gf90f4eaetrueOK57745783mirror.1trial01719.2.3-689-gf90f4eaefalseOK57805791mirror.2trial01719.2.3-689-gf90f4eaefalseOK57855790mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}57715776mirror.0trial0172026-02-22 20:09:25cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:09:27test_resync_imagec596c720-b4c1-44da-81d8-6d8beb6fbcebup+unknownremote image is not primary57805791mirror.2trial0172026-02-22 20:09:31cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:09:27' 2026-02-22T20:09:31.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:09:31.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:09:31.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:31' 2026-02-22T20:09:31.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:09:31.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:09:31.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:09:31.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:09:31.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:09:31.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:31.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:31.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test_resync_image 2026-02-22T20:09:31.817 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:31.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:31.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:31.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:09:31.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_resync_image 2026-02-22T20:09:32.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:09:32.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test_resync_image 2026-02-22T20:09:32.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:32.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:32.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:32.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_resync_image started 2026-02-22T20:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:32.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:33.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_resync_image started 2026-02-22T20:09:33.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:33.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:09:33.690 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:33.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:33.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:09:33.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:09:33.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:33.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:09:33.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:09:33.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:09:33.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:33.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:33.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:33.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:33.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:33.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:33.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:09:33.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:09:33.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:09:33.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:33.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:33.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:33.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_resync_image'] 2026-02-22T20:09:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:09:33.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_resync_image 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:33.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:09:33.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:09:33.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:09:33.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:09:33.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:33.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:33.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:33.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_resync_image 2026-02-22T20:09:33.835 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:09:33.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:09:33.836 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:09:33.837 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:09:33.838 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:09:33.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:09:33.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:09:33.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:33.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_resync_image up+stopped 2026-02-22T20:09:33.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:34.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_resync_image up+stopped '' '' 2026-02-22T20:09:34.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:34.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:34.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:34.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:09:34.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:34.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:34.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:34.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:34.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_resync_image 2026-02-22T20:09:34.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: c596c720-b4c1-44da-81d8-6d8beb6fbceb 2026-02-22T20:09:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:09:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:09:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.2 on trial017 2026-02-22T20:09:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:31 2026-02-22T20:09:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:09:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:09:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790972,"replay_state":"idle"} 2026-02-22T20:09:34.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:33 2026-02-22T20:09:34.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:09:34.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 189 .mirror.primary.c596c720-b4c1-44da-81d8-6d8beb6fbceb.2225e279-1567-408e-be18-8f4859aa9027 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:09:34.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:09:34.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:09:34.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:09:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_resync_image up+stopped '' '' 2026-02-22T20:09:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:09:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:36.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_resync_image 2026-02-22T20:09:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: c596c720-b4c1-44da-81d8-6d8beb6fbceb 2026-02-22T20:09:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:09:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:09:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.2 on trial017 2026-02-22T20:09:36.912 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:36 2026-02-22T20:09:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:09:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:09:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790972,"replay_state":"idle"} 2026-02-22T20:09:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:33 2026-02-22T20:09:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:09:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 189 .mirror.primary.c596c720-b4c1-44da-81d8-6d8beb6fbceb.2225e279-1567-408e-be18-8f4859aa9027 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:09:36.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:09:36.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:09:36.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:09:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:09:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:09:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:09:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:09:36.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:09:36.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:09:36.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.2 on trial017 2026-02-22T20:09:36.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:09:36.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:09:36.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:36' 2026-02-22T20:09:36.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_resync_image up+stopped '2026-02-22 20:09:36' 2026-02-22T20:09:36.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:36.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:36.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:36.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:09:36.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:09:36' 2026-02-22T20:09:36.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:36.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:36.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:09:36.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK257715776mirror.0trial01719.2.3-689-gf90f4eaetrueOK57745783mirror.1trial01719.2.3-689-gf90f4eaefalseOK57805791mirror.2trial01719.2.3-689-gf90f4eaefalseOK57855790mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}57715776mirror.0trial0172026-02-22 20:09:25cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:09:32test_resync_imagec596c720-b4c1-44da-81d8-6d8beb6fbcebup+stoppedlocal image is primary57805791mirror.2trial0172026-02-22 20:09:36cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790972,"replay_state":"idle"}2026-02-22 20:09:33' 2026-02-22T20:09:36.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:09:36.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:36' 2026-02-22T20:09:36.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:09:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:09:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:09:36.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:09:36.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:36.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:36.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_resync_image up+replaying 2026-02-22T20:09:36.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:36.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:36.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:36.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:09:36.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:36.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:36.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:09:36.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:37.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_resync_image up+replaying '' '' 2026-02-22T20:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:37.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_resync_image 2026-02-22T20:09:38.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_resync_image: 2026-02-22T20:09:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: c596c720-b4c1-44da-81d8-6d8beb6fbceb 2026-02-22T20:09:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:09:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790972,"replay_state":"idle"} 2026-02-22T20:09:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:09:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:33 2026-02-22T20:09:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:09:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:09:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:09:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:09:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:09:36' 2026-02-22T20:09:38.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:09:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:09:38.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:09:38.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790972,"replay_state":"idle"} 2026-02-22T20:09:38.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:09:38.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:09:38.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:09:38.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:09:38.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:09:38.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:09:38.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:09:38.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:09:38.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:33' 2026-02-22T20:09:38.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_resync_image up+replaying '2026-02-22 20:09:33' 2026-02-22T20:09:38.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:38.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:38.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:38.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:09:38.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:09:33' 2026-02-22T20:09:38.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:09:38.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:09:38.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:09:38.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK244524459mirror.0trial01719.2.3-689-gf90f4eaetrueOK44564466mirror.1trial01719.2.3-689-gf90f4eaefalseOK44634474mirror.2trial01719.2.3-689-gf90f4eaefalseOK44704480mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest3958e471-d54e-4d1a-84d5-607022ebcf0eup+stoppedlocal image is primary44524459mirror.0trial0172026-02-22 20:09:37cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771790766,"remote_snapshot_timestamp":1771790766,"replay_state":"idle"}2026-02-22 20:09:25test_resync_imagec596c720-b4c1-44da-81d8-6d8beb6fbcebup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790972,"replay_state":"idle"}44564466mirror.1trial0172026-02-22 20:09:33cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:09:36' 2026-02-22T20:09:38.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:09:38.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/last_update' 2026-02-22T20:09:38.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:09:33' 2026-02-22T20:09:38.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_resync_image'\'']/state' 2026-02-22T20:09:38.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:09:38.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:09:38.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:09:38.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:09:38.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:38.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:38.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test_resync_image 2026-02-22T20:09:38.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:38.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:38.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:09:38.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_resync_image.export 2026-02-22T20:09:38.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_resync_image.export 2026-02-22T20:09:38.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:09:38.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.export 2026-02-22T20:09:38.065 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_resync_image.export 2026-02-22T20:09:38.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_resync_image.export 2026-02-22T20:09:38.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:09:38.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.export 2026-02-22T20:09:38.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.export 2026-02-22T20:09:38.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test_resync_image /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.export 2026-02-22T20:09:38.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:09:38.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test_resync_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.export 2026-02-22T20:09:38.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:09:38.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.export 2026-02-22T20:09:38.392 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_resync_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_resync_image.export 2026-02-22T20:09:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster1 mirror test_resync_image 2026-02-22T20:09:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster1 mirror test_resync_image 2026-02-22T20:09:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_resync_image 2026-02-22T20:09:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_resync_image 2026-02-22T20:09:38.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 rm mirror/test_resync_image 2026-02-22T20:09:39.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:09:39.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:39.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: disable mirror while daemon is stopped' 2026-02-22T20:09:39.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:09:39.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:09:39.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:09:39 TEST: disable mirror while daemon is stopped 2026-02-22T20:09:39.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:09:39 TEST: disable mirror while daemon is stopped 2026-02-22T20:09:39.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirrors cluster1 2026-02-22T20:09:39.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:39.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:39.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:09:39.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:39.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:0 '' 2026-02-22T20:09:39.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:09:39.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:39.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:39.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:0 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:0 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:09:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:09:39.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:09:39.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:09:39.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=50543 2026-02-22T20:09:39.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 50543 ']' 2026-02-22T20:09:39.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 50543 2026-02-22T20:09:39.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:39.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:39.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:39.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50543 '$2 == pid {print; exit 1}' 2026-02-22T20:09:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 50543 0.5 0.0 775136 39180 ? Ssl 20:09 0:00 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:39.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:40.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:40.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50543 '$2 == pid {print; exit 1}' 2026-02-22T20:09:40.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:40.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:40.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50543 '$2 == pid {print; exit 1}' 2026-02-22T20:09:40.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:09:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:09:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:09:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:09:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:40.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:40.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:09:40.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:09:40.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:09:40.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-22T20:09:40.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:09:40.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:40.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:40.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:09:40.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:09:40.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:40.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:40.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:40.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:40.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:40.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:40.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:40.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:40.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:40.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:09:40.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:09:40.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:09:40.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:0 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:40.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:09:40.740 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:09:40.741 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:09:40.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:40.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:1 '' 2026-02-22T20:09:40.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:09:40.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:40.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:40.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:1 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:1 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:09:40.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:09:40.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:09:40.755 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:09:40.756 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=50571 2026-02-22T20:09:40.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 50571 ']' 2026-02-22T20:09:40.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 50571 2026-02-22T20:09:40.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:40.757 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:40.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:40.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50571 '$2 == pid {print; exit 1}' 2026-02-22T20:09:40.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 50571 0.4 0.0 701364 35492 ? Ssl 20:09 0:00 rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:40.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:40.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:41.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:41.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50571 '$2 == pid {print; exit 1}' 2026-02-22T20:09:41.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:41.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:41.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50571 '$2 == pid {print; exit 1}' 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:41.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:09:41.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:09:41.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:41.815 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:41.816 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:09:41.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:09:41.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:09:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:1 2026-02-22T20:09:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:09:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:09:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:09:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:41.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:09:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:09:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:41.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:09:41.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:09:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:09:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:2 '' 2026-02-22T20:09:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:09:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:2 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:2 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:09:41.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:09:41.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:09:41.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:09:41.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:09:41.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=50615 2026-02-22T20:09:41.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 50615 ']' 2026-02-22T20:09:41.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 50615 2026-02-22T20:09:41.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:41.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:41.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:41.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50615 '$2 == pid {print; exit 1}' 2026-02-22T20:09:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 50615 0.6 0.0 775132 40896 ? Ssl 20:09 0:00 rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:41.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:41.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:42.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:42.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50615 '$2 == pid {print; exit 1}' 2026-02-22T20:09:42.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:42.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:42.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50615 '$2 == pid {print; exit 1}' 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:09:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:42.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:09:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:09:42.920 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:09:42.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-22T20:09:42.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:09:42.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:42.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:42.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:09:42.934 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:09:42.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:2 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:09:42.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:09:42.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:09:42.949 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:3 '' 2026-02-22T20:09:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:09:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:42.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:42.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:3 2026-02-22T20:09:42.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:3 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:09:42.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:09:42.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:09:42.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:09:42.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:09:42.964 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=50674 2026-02-22T20:09:42.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 50674 ']' 2026-02-22T20:09:42.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 50674 2026-02-22T20:09:42.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:42.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:42.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:42.966 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50674 '$2 == pid {print; exit 1}' 2026-02-22T20:09:42.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 50674 0.4 0.0 701368 36692 ? Ssl 20:09 0:00 rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:42.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:42.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:43.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:43.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50674 '$2 == pid {print; exit 1}' 2026-02-22T20:09:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:43.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:43.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=50674 '$2 == pid {print; exit 1}' 2026-02-22T20:09:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:09:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:09:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:09:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:09:44.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:44.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:44.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:44.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:44.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:09:44.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:44.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:44.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:09:44.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:44.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:09:44.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:09:44.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-22T20:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:09:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:09:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:09:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:09:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:09:44.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:09:44.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:3 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:09:44.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:09:44.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:09:44.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirrors cluster2 2026-02-22T20:09:44.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:44.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:44.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:09:44.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:44.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster2:0 '' 2026-02-22T20:09:44.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:0 2026-02-22T20:09:44.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:44.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:44.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster2:0 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2:0 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster2 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster2 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:09:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2026-02-22T20:09:44.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-22T20:09:44.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-22T20:09:44.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=25210 2026-02-22T20:09:44.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 25210 ']' 2026-02-22T20:09:44.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 25210 2026-02-22T20:09:44.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:44.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:44.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:44.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25210 '$2 == pid {print; exit 1}' 2026-02-22T20:09:44.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 25210 0.3 0.0 773088 57084 ? Ssl 19:54 0:03 rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:44.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:45.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:45.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25210 '$2 == pid {print; exit 1}' 2026-02-22T20:09:45.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:45.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:45.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25210 '$2 == pid {print; exit 1}' 2026-02-22T20:09:45.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:0 cluster1 2026-02-22T20:09:45.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:0 2026-02-22T20:09:45.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:45.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:45.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:09:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:09:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:09:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:09:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:45.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:45.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:09:45.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:45.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:09:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:0 2026-02-22T20:09:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:09:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:09:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:45.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:09:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:09:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:09:45.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:45.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:09:45.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster2:0 2026-02-22T20:09:45.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:09:45.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:45.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:09:45.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:09:45.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:45.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:45.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:45.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:45.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:09:45.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:45.131 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:45.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:09:45.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:09:45.132 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2026-02-22T20:09:45.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-22T20:09:45.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-22T20:09:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster2:1 '' 2026-02-22T20:09:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:1 2026-02-22T20:09:45.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:45.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:45.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster2:1 2026-02-22T20:09:45.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2:1 2026-02-22T20:09:45.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:45.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:09:45.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:09:45.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:45.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:45.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:45.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:45.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:45.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster2 2026-02-22T20:09:45.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster2 2026-02-22T20:09:45.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:09:45.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:09:45.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2026-02-22T20:09:45.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-22T20:09:45.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-22T20:09:45.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=25238 2026-02-22T20:09:45.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 25238 ']' 2026-02-22T20:09:45.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 25238 2026-02-22T20:09:45.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:45.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:45.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:45.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25238 '$2 == pid {print; exit 1}' 2026-02-22T20:09:45.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 25238 0.1 0.0 775132 54972 ? Ssl 19:54 0:01 rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:45.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:45.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:46.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:46.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25238 '$2 == pid {print; exit 1}' 2026-02-22T20:09:46.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:46.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:46.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25238 '$2 == pid {print; exit 1}' 2026-02-22T20:09:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:1 cluster1 2026-02-22T20:09:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:1 2026-02-22T20:09:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:46.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:09:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:09:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:09:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:09:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:46.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:46.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:09:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:46.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:09:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:1 2026-02-22T20:09:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:09:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:09:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:46.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:46.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:46.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:46.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:09:46.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:09:46.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:46.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:46.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:09:46.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:46.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:09:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster2:1 2026-02-22T20:09:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:09:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:09:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:09:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:09:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:09:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:09:46.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2026-02-22T20:09:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-22T20:09:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-22T20:09:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:46.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster2:2 '' 2026-02-22T20:09:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:2 2026-02-22T20:09:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster2:2 2026-02-22T20:09:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2:2 2026-02-22T20:09:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:2 cluster instance 2026-02-22T20:09:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:2 2026-02-22T20:09:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-22T20:09:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster2 2026-02-22T20:09:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster2 2026-02-22T20:09:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:09:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:09:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2026-02-22T20:09:46.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-22T20:09:46.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-22T20:09:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=25298 2026-02-22T20:09:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 25298 ']' 2026-02-22T20:09:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 25298 2026-02-22T20:09:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:46.257 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25298 '$2 == pid {print; exit 1}' 2026-02-22T20:09:46.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 25298 0.1 0.0 699316 48872 ? Ssl 19:54 0:01 rbd-mirror --cluster cluster2 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:46.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:46.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:47.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25298 '$2 == pid {print; exit 1}' 2026-02-22T20:09:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:47.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25298 '$2 == pid {print; exit 1}' 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:2 cluster1 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:2 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:09:47.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:47.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-22T20:09:47.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:09:47.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:09:47.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-22T20:09:47.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:2 2026-02-22T20:09:47.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:47.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-22T20:09:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:09:47.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster2:2 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:2 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:47.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:09:47.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:09:47.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2026-02-22T20:09:47.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-22T20:09:47.336 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-22T20:09:47.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:47.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster2:3 '' 2026-02-22T20:09:47.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:3 2026-02-22T20:09:47.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:09:47.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:47.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:09:47.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster2:3 2026-02-22T20:09:47.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2:3 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:3 cluster instance 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:3 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster2 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster2 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:09:47.339 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2026-02-22T20:09:47.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-22T20:09:47.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-22T20:09:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=25343 2026-02-22T20:09:47.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 25343 ']' 2026-02-22T20:09:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 25343 2026-02-22T20:09:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:47.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:09:47.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:47.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25343 '$2 == pid {print; exit 1}' 2026-02-22T20:09:47.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 25343 0.1 0.0 701368 48988 ? Ssl 19:54 0:01 rbd-mirror --cluster cluster2 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:47.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:09:47.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:48.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:48.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25343 '$2 == pid {print; exit 1}' 2026-02-22T20:09:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:09:48.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:09:48.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=25343 '$2 == pid {print; exit 1}' 2026-02-22T20:09:48.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:3 cluster1 2026-02-22T20:09:48.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:3 2026-02-22T20:09:48.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:48.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:48.396 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-22T20:09:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:09:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:09:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:09:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:09:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:09:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:48.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-22T20:09:48.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:09:48.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:09:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-22T20:09:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:3 2026-02-22T20:09:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:09:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-22T20:09:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:09:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:09:48.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:48.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-22T20:09:48.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:09:48.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:09:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster2:3 2026-02-22T20:09:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:3 2026-02-22T20:09:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:09:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-22T20:09:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:09:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:09:48.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:09:48.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:09:48.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:48.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:09:48.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:09:48.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:09:48.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:09:48.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:09:48.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2026-02-22T20:09:48.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-22T20:09:48.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-22T20:09:48.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:09:48.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ disable_mirror cluster2 mirror test 2026-02-22T20:09:48.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:48.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:48.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:09:48.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test 2026-02-22T20:09:49.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring disabled 2026-02-22T20:09:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z '' ']' 2026-02-22T20:09:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test present 2026-02-22T20:09:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:09:49.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:09:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:09:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:09:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:09:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:09:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:09:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:09:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:09:49.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:09:49.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=10fa675d7aa6 2026-02-22T20:09:49.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 10fa675d7aa6 2026-02-22T20:09:49.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z '' -o '' = 10fa675d7aa6 2026-02-22T20:09:49.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:09:49.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = present 2026-02-22T20:09:49.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirrors cluster1 2026-02-22T20:09:49.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:49.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:09:49.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:49.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:0 2026-02-22T20:09:49.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:09:49.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:09:49.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:09:49.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T20:09:49.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:09:49.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:09:49.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:09:49.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T20:09:49.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:09:49.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:09:49.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:49.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T20:09:49.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:09:49.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:49.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:1 2026-02-22T20:09:49.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:09:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:09:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:09:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T20:09:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:09:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:09:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:09:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T20:09:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:09:49.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:09:49.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:49.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T20:09:49.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:09:49.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:49.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:2 2026-02-22T20:09:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:09:49.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:09:49.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:09:49.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T20:09:49.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:09:49.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:09:49.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:09:49.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T20:09:49.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:09:49.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:09:49.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:49.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T20:09:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:09:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:49.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:49.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:09:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:3 2026-02-22T20:09:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:09:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:09:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:09:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T20:09:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:09:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:09:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:09:49.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T20:09:49.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:09:49.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:09:49.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:09:49.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T20:09:49.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:09:49.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:49.265 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:09:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test deleted 2026-02-22T20:09:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:09:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T20:09:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:09:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:09:49.282 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:09:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:09:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:09:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:09:49.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:09:49.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id=10fa675d7aa6 2026-02-22T20:09:49.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:09:49.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:09:49.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test deleted 10fa675d7aa6 2026-02-22T20:09:49.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:49.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:49.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:09:49.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:09:49.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=10fa675d7aa6 2026-02-22T20:09:49.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:09:49.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:09:49.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:09:49.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:49.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:09:49.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:09:49.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:09:49.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:09:49.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=10fa675d7aa6 2026-02-22T20:09:49.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 10fa675d7aa6 2026-02-22T20:09:49.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 10fa675d7aa6 -o 10fa675d7aa6 = 10fa675d7aa6 2026-02-22T20:09:49.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:09:49.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = present 2026-02-22T20:09:49.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:09:49.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:50.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test deleted 10fa675d7aa6 2026-02-22T20:09:50.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:50.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:50.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:09:50.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:09:50.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=10fa675d7aa6 2026-02-22T20:09:50.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:09:50.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:09:50.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:09:50.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:50.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:09:50.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:09:50.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:09:50.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:09:50.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=10fa675d7aa6 2026-02-22T20:09:50.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 10fa675d7aa6 2026-02-22T20:09:50.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 10fa675d7aa6 -o 10fa675d7aa6 = 10fa675d7aa6 2026-02-22T20:09:50.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:09:50.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = present 2026-02-22T20:09:50.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:09:50.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:09:52.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test deleted 10fa675d7aa6 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=10fa675d7aa6 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:09:52.482 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:09:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=10fa675d7aa6 2026-02-22T20:09:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 10fa675d7aa6 2026-02-22T20:09:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 10fa675d7aa6 -o 10fa675d7aa6 = 10fa675d7aa6 2026-02-22T20:09:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:09:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = present 2026-02-22T20:09:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:09:52.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 4 2026-02-22T20:09:56.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test deleted 10fa675d7aa6 2026-02-22T20:09:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:09:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:09:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=10fa675d7aa6 2026-02-22T20:09:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:09:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:09:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:09:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:09:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:09:56.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:09:56.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:09:56.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:09:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:09:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T20:09:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:09:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:09:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test 2026-02-22T20:09:56.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:09:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:09:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:09:56.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2026-02-22T20:09:57.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:09:57.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test 2026-02-22T20:09:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test': 2026-02-22T20:09:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:09:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:09:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 11514e1103e6 2026-02-22T20:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.11514e1103e6 2026-02-22T20:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 19:54:42 2026 2026-02-22T20:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 19:55:45 2026 2026-02-22T20:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 19:55:55 2026 2026-02-22T20:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 11514e1103e6 2026-02-22T20:09:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:09:57.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:09:57.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 516307d5-169a-48a7-bdf2-62efacc5d454 2026-02-22T20:09:57.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:09:57.200 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test present 2026-02-22T20:09:57.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:57.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:57.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:09:57.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=present 2026-02-22T20:09:57.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:09:57.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:09:57.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:57.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:09:57.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:57.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:09:57.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:09:57.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:09:57.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:09:57.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:09:57.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id= 2026-02-22T20:09:57.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:09:57.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:09:57.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-22T20:09:57.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:57.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:57.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:09:57.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:09:57.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:09:57.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:09:57.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:09:57.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:09:57.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:57.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:09:57.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:09:57.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:09:57.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:09:57.352 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:09:57.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:09:57.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:57.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = deleted 2026-02-22T20:09:57.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:09:57.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:09:58.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-22T20:09:58.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:09:58.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:09:58.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:09:58.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:09:58.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:09:58.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:09:58.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:09:58.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:09:58.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:09:58.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:09:58.358 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:09:58.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:09:58.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:09:58.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:09:58.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:09:58.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:09:58.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = deleted 2026-02-22T20:09:58.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:09:58.461 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:10:00.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-22T20:10:00.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:00.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:00.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:00.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:10:00.462 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:10:00.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:00.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:00.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:00.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:00.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:00.463 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:00.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:00.464 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:00.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=16dbdb78612b 2026-02-22T20:10:00.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 16dbdb78612b 2026-02-22T20:10:00.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z '' -o '' = 16dbdb78612b 2026-02-22T20:10:00.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:10:00.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = present 2026-02-22T20:10:00.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:00.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:10:00.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:10:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:10:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:00.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:01.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:10:01.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:01.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:01.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:01.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:10:01.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:10:01.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:10:01.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:10:01.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:10:01.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:10:01.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:01.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:01.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:10:01.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:10:01.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:01.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:10:01.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:01.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:10:01.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:01.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:10:01.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:01.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:01.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:01.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:01.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:01.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:10:01.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:10:01.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:01.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:01.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:01.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:01.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:10:01.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:10:01.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:10:01.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:10:01.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:10:01.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:10:01.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:10:01.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:10:01.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:01.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: non-default namespace image mirroring' 2026-02-22T20:10:01.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:10:01.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:10:01.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:10:01 TEST: non-default namespace image mirroring 2026-02-22T20:10:01.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:10:01 TEST: non-default namespace image mirroring 2026-02-22T20:10:01.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog ' - replay' 2026-02-22T20:10:01.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:10:01.576 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:10:01.577 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:10:01 - replay 2026-02-22T20:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:10:01 - replay 2026-02-22T20:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns1 test snapshot 2026-02-22T20:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:10:01.578 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror/ns1 test 2026-02-22T20:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:10:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns1/test 2026-02-22T20:10:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror/ns1 2026-02-22T20:10:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:10:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror/ns1 test snapshot 2026-02-22T20:10:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:01.825 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:01.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:01.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:10:01.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/ns1/test snapshot 2026-02-22T20:10:02.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:10:02.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/ns1/test 2026-02-22T20:10:02.201 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test': 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 1793c6bbf38e 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.1793c6bbf38e 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:10:01 2026 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:10:01 2026 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:10:01 2026 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 1793c6bbf38e 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 16d432d4-fdd1-48a6-8bab-c074737c01b5 2026-02-22T20:10:02.202 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns2 test snapshot 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror/ns2 test 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:10:02.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns2/test 2026-02-22T20:10:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror/ns2 2026-02-22T20:10:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:10:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror/ns2 test snapshot 2026-02-22T20:10:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:10:02.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/ns2/test snapshot 2026-02-22T20:10:03.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:10:03.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/ns2/test 2026-02-22T20:10:03.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test': 2026-02-22T20:10:03.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:10:03.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:10:03.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:10:03.203 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 179c63e06449 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.179c63e06449 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:10:02 2026 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:10:02 2026 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:10:02 2026 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 179c63e06449 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: eda12e8b-c4f1-4790-82cd-36d6513f123f 2026-02-22T20:10:03.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:10:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test 2026-02-22T20:10:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test started 2026-02-22T20:10:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:10:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:03.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:03.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:04.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror/ns1 test started 2026-02-22T20:10:04.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:04.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:04.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:04.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:10:04.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:10:04.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:10:04.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns1/test 2026-02-22T20:10:04.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:10:04.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:10:04.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:04.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:04.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:10:04.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:10:04.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:04.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:10:04.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:04.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:10:04.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:04.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2026-02-22T20:10:04.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:10:04.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:04.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:10:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:10:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:04.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:04.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:04.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:04.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:04.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:04.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:10:04.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:10:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:04.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2026-02-22T20:10:04.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:10:04.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:10:04.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:10:04.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:10:04.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:10:04.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:10:04.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:10:04.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror/ns2 test 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror/ns2 test started 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror/ns2 test started 2026-02-22T20:10:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:10:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:10:05.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:10:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns2/test 2026-02-22T20:10:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:10:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:10:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:05.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:05.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:10:05.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:10:05.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:05.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:10:05.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:05.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:10:05.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns2/test 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:05.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:10:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:10:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:05.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:10:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:10:05.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:05.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:05.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:05.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:05.310 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns2/test 2026-02-22T20:10:05.377 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:10:05.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:10:05.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:10:05.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:10:05.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:10:05.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:10:05.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:10:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror/ns1 test 100 2026-02-22T20:10:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:10:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:10:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:10:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:10:05.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:10:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:10:05.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.19 bytes/sec: 24 MiB/s 2026-02-22T20:10:05.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror/ns2 test 100 2026-02-22T20:10:05.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:05.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:05.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:05.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:10:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:10:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:10:05.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/ns2/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:10:05.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:10:05.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:10:05.488 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.19 bytes/sec: 24 MiB/s 2026-02-22T20:10:05.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 test 2026-02-22T20:10:05.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:10:05.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:05.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:05.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:05.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:10:05.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:10:05.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror/ns1 test 2026-02-22T20:10:05.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:05.497 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:05.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:05.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/ns1/test 2026-02-22T20:10:06.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 188 2026-02-22T20:10:06.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror/ns1 test 2026-02-22T20:10:06.182 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:10:06.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:06.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:06.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:06.183 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror/ns1-test.status 2026-02-22T20:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror/ns1-test.status 2026-02-22T20:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:06.184 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test.status 2026-02-22T20:10:06.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror/ns1-test.status 2026-02-22T20:10:06.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror/ns1-test.status 2026-02-22T20:10:06.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test.status 2026-02-22T20:10:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror/ns1 test /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test.status 2026-02-22T20:10:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test.status 2026-02-22T20:10:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/ns1/test --format xml 2026-02-22T20:10:06.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:10:06.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:10:06.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=188 2026-02-22T20:10:06.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:10:06.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:10:06.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:10:06.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror/ns1 test /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test.status 2026-02-22T20:10:06.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:06.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:06.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:06.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test.status 2026-02-22T20:10:06.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/ns1/test --format xml 2026-02-22T20:10:06.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:10:06.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:10:06.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=186 2026-02-22T20:10:06.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 188 = 186 2026-02-22T20:10:06.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:10:06.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:10:06.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror/ns1 test /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test.status 2026-02-22T20:10:06.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:06.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:06.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:06.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test.status 2026-02-22T20:10:06.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/ns1/test --format xml 2026-02-22T20:10:06.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:10:06.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:10:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=188 2026-02-22T20:10:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 188 = 188 2026-02-22T20:10:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns2 test 2026-02-22T20:10:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:10:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:10:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:10:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror/ns2 test 2026-02-22T20:10:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/ns2/test 2026-02-22T20:10:07.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 189 2026-02-22T20:10:07.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror/ns2 test 2026-02-22T20:10:07.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:10:07.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:07.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:07.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:07.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror/ns2-test.status 2026-02-22T20:10:07.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror/ns2-test.status 2026-02-22T20:10:07.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:07.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror_ns2-test.status 2026-02-22T20:10:07.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror/ns2-test.status 2026-02-22T20:10:07.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror/ns2-test.status 2026-02-22T20:10:07.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:07.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns2-test.status 2026-02-22T20:10:07.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror/ns2 test /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns2-test.status 2026-02-22T20:10:07.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:07.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:07.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:07.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror_ns2-test.status 2026-02-22T20:10:07.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/ns2/test --format xml 2026-02-22T20:10:07.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:10:07.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:10:07.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=189 2026-02-22T20:10:07.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:10:07.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:10:07.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:10:07.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror/ns2 test /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns2-test.status 2026-02-22T20:10:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns2-test.status 2026-02-22T20:10:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/ns2/test --format xml 2026-02-22T20:10:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:10:07.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:10:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=187 2026-02-22T20:10:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 189 = 187 2026-02-22T20:10:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:10:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:10:07.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror/ns2 test /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns2-test.status 2026-02-22T20:10:07.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:07.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:07.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:07.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns2-test.status 2026-02-22T20:10:07.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/ns2/test --format xml 2026-02-22T20:10:07.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:10:07.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:10:07.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=189 2026-02-22T20:10:07.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 189 = 189 2026-02-22T20:10:07.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:07.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns1 test 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test up+replaying 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:07.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:08.894 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test up+replaying '' '' 2026-02-22T20:10:08.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:08.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:08.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:08.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:08.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:08.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:08.895 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:08.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:08.896 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns1/test 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 16d432d4-fdd1-48a6-8bab-c074737c01b5 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791002,"replay_state":"idle"} 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:10:04 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:10:08.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:10:08.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:10:08.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:10:08.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791002,"replay_state":"idle"} 2026-02-22T20:10:08.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:10:08.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:08.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:10:08.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:08.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:10:08.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:10:08.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:10:08.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:10:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:04' 2026-02-22T20:10:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test up+replaying '2026-02-22 20:10:04' 2026-02-22T20:10:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:10:04' 2026-02-22T20:10:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml 2026-02-22T20:10:08.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING158415851mirror.0trial01719.2.3-689-gf90f4eaetrueOK58455858mirror.1trial01719.2.3-689-gf90f4eaefalseOK58525864mirror.2trial01719.2.3-689-gf90f4eaefalseOK58575869mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest16d432d4-fdd1-48a6-8bab-c074737c01b5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791002,"replay_state":"idle"}58415851mirror.0trial0172026-02-22 20:10:04cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:10:08.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:10:08.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:10:08.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:04' 2026-02-22T20:10:08.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:10:08.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:10:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:10:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:08.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns2 test 2026-02-22T20:10:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:10:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns2 test up+replaying 2026-02-22T20:10:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:08.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:08.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:08.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:08.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:08.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror/ns2 test up+replaying '' '' 2026-02-22T20:10:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:09.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:09.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns2/test 2026-02-22T20:10:10.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:10:10.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: eda12e8b-c4f1-4790-82cd-36d6513f123f 2026-02-22T20:10:10.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:10:10.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791003,"replay_state":"idle"} 2026-02-22T20:10:10.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:10:10.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:10:05 2026-02-22T20:10:10.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:10:10.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:10:10.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:10:10.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:10:10.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:10:10.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:10:10.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:10:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:10:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791003,"replay_state":"idle"} 2026-02-22T20:10:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:10:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:10.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:10:10.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:10.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:10:10.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:10:10.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:10:10.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:10:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:05' 2026-02-22T20:10:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns2 test up+replaying '2026-02-22 20:10:05' 2026-02-22T20:10:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:10:05' 2026-02-22T20:10:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns2 --verbose --format xml 2026-02-22T20:10:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING158415851mirror.0trial01719.2.3-689-gf90f4eaetrueOK58455858mirror.1trial01719.2.3-689-gf90f4eaefalseOK58525864mirror.2trial01719.2.3-689-gf90f4eaefalseOK58575869mirror.3trial01719.2.3-689-gf90f4eaefalseOKtesteda12e8b-c4f1-4790-82cd-36d6513f123fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791003,"replay_state":"idle"}58415851mirror.0trial0172026-02-22 20:10:05cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:10:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:10:10.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:10:10.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:05' 2026-02-22T20:10:10.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:10:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:10:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:10:10.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:10.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:10.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:10.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror/ns1 test 2026-02-22T20:10:10.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:10.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:10.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:10:10.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror/ns1-test.export 2026-02-22T20:10:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror/ns1-test.export 2026-02-22T20:10:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:10.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test.export 2026-02-22T20:10:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror/ns1-test.export 2026-02-22T20:10:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror/ns1-test.export 2026-02-22T20:10:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:10.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test.export 2026-02-22T20:10:10.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test.export 2026-02-22T20:10:10.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/ns1/test /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test.export 2026-02-22T20:10:10.199 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:10.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/ns1/test /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test.export 2026-02-22T20:10:10.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:10.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test.export 2026-02-22T20:10:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test.export 2026-02-22T20:10:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror/ns2 test 2026-02-22T20:10:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:10:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror/ns2-test.export 2026-02-22T20:10:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror/ns2-test.export 2026-02-22T20:10:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:10.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror_ns2-test.export 2026-02-22T20:10:10.449 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror/ns2-test.export 2026-02-22T20:10:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror/ns2-test.export 2026-02-22T20:10:10.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns2-test.export 2026-02-22T20:10:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns2-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns2-test.export 2026-02-22T20:10:10.452 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/ns2/test /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns2-test.export 2026-02-22T20:10:10.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:10.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/ns2/test /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns2-test.export 2026-02-22T20:10:10.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:10.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns2-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns2-test.export 2026-02-22T20:10:10.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns2-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns2-test.export 2026-02-22T20:10:10.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:10.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog ' - disable mirroring / delete image' 2026-02-22T20:10:10.826 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:10:10.827 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:10:10.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:10:10 - disable mirroring / delete image 2026-02-22T20:10:10.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:10:10 - disable mirroring / delete image 2026-02-22T20:10:10.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror/ns1 test 2026-02-22T20:10:10.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:10.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:10.828 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:10.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:10.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:10:10.829 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror/ns1 test 2026-02-22T20:10:10.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:10.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:10.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:10.830 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test 2026-02-22T20:10:10.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test 2026-02-22T20:10:13.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:10:13.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ disable_mirror cluster2 mirror/ns2 test 2026-02-22T20:10:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:13.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/ns2/test 2026-02-22T20:10:14.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring disabled 2026-02-22T20:10:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror/ns1 test deleted 2026-02-22T20:10:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:14.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T20:10:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:10:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror/ns1 test 2026-02-22T20:10:14.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:14.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror/ns1 2026-02-22T20:10:14.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:14.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test 2026-02-22T20:10:14.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:15.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:10:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id= 2026-02-22T20:10:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:10:15.113 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror/ns1 test deleted '' 2026-02-22T20:10:15.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:15.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:15.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:15.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:10:15.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:10:15.114 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:15.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:15.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror/ns1 test 2026-02-22T20:10:15.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:15.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror/ns1 2026-02-22T20:10:15.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:15.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test 2026-02-22T20:10:15.115 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:15.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:10:15.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:10:15.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:15.141 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T20:10:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror/ns2 test deleted 2026-02-22T20:10:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T20:10:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:10:15.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:15.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:15.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror/ns2 test 2026-02-22T20:10:15.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:15.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror/ns2 2026-02-22T20:10:15.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:15.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/ns2/test 2026-02-22T20:10:15.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:15.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:10:15.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id= 2026-02-22T20:10:15.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:15.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:10:15.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror/ns2 test deleted '' 2026-02-22T20:10:15.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:15.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:15.267 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:15.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:10:15.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:10:15.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:15.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:15.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror/ns2 test 2026-02-22T20:10:15.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:15.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror/ns2 2026-02-22T20:10:15.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:15.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/ns2/test 2026-02-22T20:10:15.268 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:15.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T20:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror/ns2 test 2026-02-22T20:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:15.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:10:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror/ns2 test 2026-02-22T20:10:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns2 2026-02-22T20:10:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:15.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns2/test 2026-02-22T20:10:15.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/ns2/test 2026-02-22T20:10:15.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:10:15.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:15.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog ' - data pool' 2026-02-22T20:10:15.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:10:15.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:10:15.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:10:15 - data pool 2026-02-22T20:10:15.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:10:15 - data pool 2026-02-22T20:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ dp_image=test_data_pool 2026-02-22T20:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns1 test_data_pool snapshot 128 --data-pool mirror_parent 2026-02-22T20:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:10:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror/ns1 test_data_pool 128 --data-pool mirror_parent 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 128 ']' 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=128 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:15.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/ns1/test_data_pool 2026-02-22T20:10:15.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror/ns1 2026-02-22T20:10:15.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:15.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:10:15.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:15.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror/ns1 test_data_pool snapshot 2026-02-22T20:10:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:10:15.448 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/ns1/test_data_pool snapshot 2026-02-22T20:10:15.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:10:15.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/ns1/test_data_pool 2026-02-22T20:10:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_data_pool': 2026-02-22T20:10:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:10:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:10:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:10:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 17cc4c41f76e 2026-02-22T20:10:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: data_pool: mirror_parent 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.3.17cc4c41f76e 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling, data-pool 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:10:15 2026 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:10:15 2026 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:10:15 2026 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 17cc4c41f76e 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:10:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: e8b0114f-5c1c-44f9-9322-ddbb72ee70ca 2026-02-22T20:10:16.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:10:16.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_data_pool cluster2 mirror/ns1 test_data_pool 2026-02-22T20:10:16.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:10:16.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror/ns1 2026-02-22T20:10:16.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_data_pool 2026-02-22T20:10:16.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 info mirror/ns1/test_data_pool 2026-02-22T20:10:16.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ awk '$1 == "data_pool:" {print $2}' 2026-02-22T20:10:16.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ data_pool=mirror_parent 2026-02-22T20:10:16.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test mirror_parent = mirror_parent 2026-02-22T20:10:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test_data_pool 2026-02-22T20:10:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test_data_pool started 2026-02-22T20:10:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:16.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:10:16.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:16.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:16.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:17.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror/ns1 test_data_pool started 2026-02-22T20:10:17.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:17.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:17.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:17.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:10:17.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:10:17.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:10:17.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns1/test_data_pool 2026-02-22T20:10:17.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:10:17.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:10:17.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:17.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:17.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:10:17.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:10:17.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:17.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:10:17.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:17.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2026-02-22T20:10:17.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:17.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:17.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:17.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:17.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:10:17.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:10:17.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:17.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:17.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:17.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:17.062 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-22T20:10:17.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-22T20:10:17.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:10:17.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:10:17.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:17.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2026-02-22T20:10:17.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:10:17.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:17.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:17.123 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:10:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:10:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:10:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:10:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:10:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:10:17.124 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:17.125 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:17.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:10:17.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:10:17.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:10:17.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:10:17.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:10:17.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:10:17.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-22T20:10:17.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-22T20:10:17.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:10:17.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:10:17.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:17.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test_data_pool 2026-02-22T20:10:17.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:10:17.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:17.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:17.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:10:17.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:17.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:10:17.198 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:10:17.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:10:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:10:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:10:17.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-22T20:10:17.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-22T20:10:17.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:10:17.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test_data_pool 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:10:17.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:10:17.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:17.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:17.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:10:17.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:10:17.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:10:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:10:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:10:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:10:17.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-22T20:10:17.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2026-02-22T20:10:17.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:10:17.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:10:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:17.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T20:10:18.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:18.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:10:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2026-02-22T20:10:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:10:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:18.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:18.333 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:18.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:10:18.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:18.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:18.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:18.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:18.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:18.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:18.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:18.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:10:18.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:10:18.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:18.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:18.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:18.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:18.344 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2026-02-22T20:10:18.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:10:18.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:10:18.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:10:18.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:10:18.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:10:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:10:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:10:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_data_pool cluster1 mirror/ns1 test_data_pool 2026-02-22T20:10:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror/ns1 2026-02-22T20:10:18.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_data_pool 2026-02-22T20:10:18.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 info mirror/ns1/test_data_pool 2026-02-22T20:10:18.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ awk '$1 == "data_pool:" {print $2}' 2026-02-22T20:10:18.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ data_pool=mirror_parent 2026-02-22T20:10:18.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test mirror_parent = mirror_parent 2026-02-22T20:10:18.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror/ns1 test_data_pool 100 2026-02-22T20:10:18.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:18.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:18.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:18.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:10:18.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:10:18.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:18.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:10:18.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:10:18.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:10:18.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:10:18.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 8333.58 bytes/sec: 33 MiB/s 2026-02-22T20:10:18.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 test_data_pool 2026-02-22T20:10:18.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:10:18.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:18.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:18.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:18.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:10:18.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:10:18.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror/ns1 test_data_pool 2026-02-22T20:10:18.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:18.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:18.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:18.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/ns1/test_data_pool 2026-02-22T20:10:19.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 17 2026-02-22T20:10:19.172 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror/ns1 test_data_pool 2026-02-22T20:10:19.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:10:19.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:19.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:19.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:19.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror/ns1-test_data_pool.status 2026-02-22T20:10:19.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror/ns1-test_data_pool.status 2026-02-22T20:10:19.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:19.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test_data_pool.status 2026-02-22T20:10:19.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror/ns1-test_data_pool.status 2026-02-22T20:10:19.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror/ns1-test_data_pool.status 2026-02-22T20:10:19.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:19.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test_data_pool.status 2026-02-22T20:10:19.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror/ns1 test_data_pool /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test_data_pool.status 2026-02-22T20:10:19.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:19.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:19.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:19.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test_data_pool.status 2026-02-22T20:10:19.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/ns1/test_data_pool --format xml 2026-02-22T20:10:19.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:10:19.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:10:19.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=17 2026-02-22T20:10:19.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:10:19.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:10:19.218 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:10:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror/ns1 test_data_pool /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test_data_pool.status 2026-02-22T20:10:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test_data_pool.status 2026-02-22T20:10:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/ns1/test_data_pool --format xml 2026-02-22T20:10:19.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:10:19.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:10:19.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=16 2026-02-22T20:10:19.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 17 = 16 2026-02-22T20:10:19.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:10:19.453 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:10:19.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror/ns1 test_data_pool /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test_data_pool.status 2026-02-22T20:10:19.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:19.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:19.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:19.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test_data_pool.status 2026-02-22T20:10:19.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/ns1/test_data_pool --format xml 2026-02-22T20:10:19.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:10:19.886 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:10:19.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=17 2026-02-22T20:10:19.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 17 = 17 2026-02-22T20:10:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns1 test_data_pool 2026-02-22T20:10:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:10:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test_data_pool up+replaying 2026-02-22T20:10:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:19.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:20.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test_data_pool up+replaying '' '' 2026-02-22T20:10:20.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:20.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:20.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:20.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:20.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:20.892 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:20.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:20.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:20.893 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns1/test_data_pool 2026-02-22T20:10:20.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_data_pool: 2026-02-22T20:10:20.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: e8b0114f-5c1c-44f9-9322-ddbb72ee70ca 2026-02-22T20:10:20.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:10:20.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771791015,"remote_snapshot_timestamp":1771791015,"replay_state":"idle"} 2026-02-22T20:10:20.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:10:20.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:10:19 2026-02-22T20:10:20.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:10:20.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:10:20.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:10:20.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:10:20.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:10:20.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:10:20.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:10:20.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:10:20.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771791015,"remote_snapshot_timestamp":1771791015,"replay_state":"idle"} 2026-02-22T20:10:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:10:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:20.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:10:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:10:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:10:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:10:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:10:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:19' 2026-02-22T20:10:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test_data_pool up+replaying '2026-02-22 20:10:19' 2026-02-22T20:10:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:10:19' 2026-02-22T20:10:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml 2026-02-22T20:10:20.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING158415851mirror.0trial01719.2.3-689-gf90f4eaetrueOK58455858mirror.1trial01719.2.3-689-gf90f4eaefalseOK58525864mirror.2trial01719.2.3-689-gf90f4eaefalseOK58575869mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_data_poole8b0114f-5c1c-44f9-9322-ddbb72ee70caup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771791015,"remote_snapshot_timestamp":1771791015,"replay_state":"idle"}58415851mirror.0trial0172026-02-22 20:10:19cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:10:20.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:10:20.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/last_update' 2026-02-22T20:10:20.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:19' 2026-02-22T20:10:20.978 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/state' 2026-02-22T20:10:20.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:10:20.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:20.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:10:20.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:20.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:20.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:20.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror/ns1 test_data_pool 2026-02-22T20:10:20.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:20.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:20.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:10:20.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror/ns1-test_data_pool.export 2026-02-22T20:10:20.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror/ns1-test_data_pool.export 2026-02-22T20:10:20.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:20.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test_data_pool.export 2026-02-22T20:10:20.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror/ns1-test_data_pool.export 2026-02-22T20:10:20.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror/ns1-test_data_pool.export 2026-02-22T20:10:20.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:20.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test_data_pool.export 2026-02-22T20:10:20.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test_data_pool.export 2026-02-22T20:10:20.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/ns1/test_data_pool /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test_data_pool.export 2026-02-22T20:10:21.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:21.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/ns1/test_data_pool /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test_data_pool.export 2026-02-22T20:10:21.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:21.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test_data_pool.export 2026-02-22T20:10:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.bhaDYZBKee/cluster1-mirror_ns1-test_data_pool.export 2026-02-22T20:10:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror/ns1 test_data_pool 2026-02-22T20:10:21.348 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:21.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:21.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:21.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:21.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:10:21.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror/ns1 test_data_pool 2026-02-22T20:10:21.349 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:21.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror/ns1 2026-02-22T20:10:21.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_data_pool 2026-02-22T20:10:21.350 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test_data_pool 2026-02-22T20:10:21.378 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test_data_pool 2026-02-22T20:10:23.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:10:23.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:23.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: simple image resync' 2026-02-22T20:10:23.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:10:23.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:10:23.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:10:23 TEST: simple image resync 2026-02-22T20:10:23.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:10:23 TEST: simple image resync 2026-02-22T20:10:23.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ request_resync_image cluster1 mirror test image_id 2026-02-22T20:10:23.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:23.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id_var_name=image_id 2026-02-22T20:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:23.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:23.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'image_id='\''16dbdb78612b'\''' 2026-02-22T20:10:23.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ image_id=16dbdb78612b 2026-02-22T20:10:23.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'test -n "$image_id"' 2026-02-22T20:10:23.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -n 16dbdb78612b 2026-02-22T20:10:23.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2026-02-22T20:10:23.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Flagged image for resync from primary 2026-02-22T20:10:23.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test deleted 16dbdb78612b 2026-02-22T20:10:23.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:23.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:23.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:23.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T20:10:23.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=16dbdb78612b 2026-02-22T20:10:23.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:23.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 16dbdb78612b 2026-02-22T20:10:23.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:23.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:10:23.380 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test deleted 16dbdb78612b 2026-02-22T20:10:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:10:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=16dbdb78612b 2026-02-22T20:10:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:23.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:23.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:23.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:23.382 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:23.404 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:10:23.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:10:23.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:23.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T20:10:23.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:23.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test present 2026-02-22T20:10:23.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:23.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:23.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:23.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=present 2026-02-22T20:10:23.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:10:23.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:23.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:23.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:23.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:23.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:23.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:23.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:23.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:23.430 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:10:23.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id= 2026-02-22T20:10:23.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:23.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:23.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:23.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:23.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:23.557 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:10:23.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:10:23.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:23.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = deleted 2026-02-22T20:10:23.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:23.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:24.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test present '' 2026-02-22T20:10:24.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:24.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:24.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:24.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:10:24.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:10:24.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:24.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:24.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:24.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:24.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:24.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:24.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:24.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:24.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=16dbbc115242 2026-02-22T20:10:24.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 16dbbc115242 2026-02-22T20:10:24.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z '' -o '' = 16dbbc115242 2026-02-22T20:10:24.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:10:24.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = present 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:24.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:25.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:10:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:10:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:10:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:10:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:10:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:10:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:10:25.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:25.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:25.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:10:25.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:10:25.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:25.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:10:25.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:25.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:10:25.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:25.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:10:25.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:10:25.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:25.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:25.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:25.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:25.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:10:25.607 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:25.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:25.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:25.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:25.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:10:25.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:10:25.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:10:25.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:10:25.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:10:25.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:10:25.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:10:25.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:10:25.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:25.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:25.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:25.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:26.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:10:26.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:26.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:26.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:26.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:26.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:26.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:26.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:26.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:26.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:10:26.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:10:26.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 516307d5-169a-48a7-bdf2-62efacc5d454 2026-02-22T20:10:26.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:10:26.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790997,"replay_state":"idle"} 2026-02-22T20:10:26.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:10:26.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:10:24 2026-02-22T20:10:26.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:10:26.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:10:26.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:10:26.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:10:26.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:10:26.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:10:26.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:10:26.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:10:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790997,"replay_state":"idle"} 2026-02-22T20:10:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:10:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:26.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:10:26.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:26.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:10:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:10:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:10:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:10:26.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:24' 2026-02-22T20:10:26.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:10:24' 2026-02-22T20:10:26.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:26.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:26.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:26.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:26.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:10:24' 2026-02-22T20:10:26.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:26.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:26.728 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:10:26.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING158415851mirror.0trial01719.2.3-689-gf90f4eaetrueOK58455858mirror.1trial01719.2.3-689-gf90f4eaefalseOK58525864mirror.2trial01719.2.3-689-gf90f4eaefalseOK58575869mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest516307d5-169a-48a7-bdf2-62efacc5d454up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790997,"replay_state":"idle"}58415851mirror.0trial0172026-02-22 20:10:24cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:10:26.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:10:26.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:10:26.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:24' 2026-02-22T20:10:26.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:10:26.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:10:26.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:26.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:10:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:26.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:26.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:26.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test 2026-02-22T20:10:26.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:10:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.export 2026-02-22T20:10:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.export 2026-02-22T20:10:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:26.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T20:10:26.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.export 2026-02-22T20:10:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.export 2026-02-22T20:10:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T20:10:26.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:26.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:27.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:27.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:27.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:27.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:27.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z '' ']' 2026-02-22T20:10:27.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: image resync while replayer is stopped' 2026-02-22T20:10:27.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:10:27.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:10:27.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:10:27 TEST: image resync while replayer is stopped 2026-02-22T20:10:27.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:10:27 TEST: image resync while replayer is stopped 2026-02-22T20:10:27.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemons cluster1 rbd mirror stop mirror/test 2026-02-22T20:10:27.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_instance=cluster1 2026-02-22T20:10:27.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:27.154 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance=cluster1 2026-02-22T20:10:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loop_instance 2026-02-22T20:10:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:10:27.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:27.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:10:27.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:27.157 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror/test 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T20:10:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:10:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:10:27.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:10:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:10:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:27.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:27.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:27.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:27.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:27.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror/test 2026-02-22T20:10:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2026-02-22T20:10:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2026-02-22T20:10:27.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:10:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:28.235 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test stopped 2026-02-22T20:10:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:10:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:10:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:10:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:10:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:10:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:10:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:10:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:10:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:10:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:10:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:10:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:10:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:28.242 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:28.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:28.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:10:28.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:10:28.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:28.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:28.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:28.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:28.252 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:10:28.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:10:28.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:10:28.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:10:28.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:10:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:10:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ request_resync_image cluster1 mirror test image_id 2026-02-22T20:10:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id_var_name=image_id 2026-02-22T20:10:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:28.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:28.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:28.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:28.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:28.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'image_id='\''16dbbc115242'\''' 2026-02-22T20:10:28.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ image_id=16dbbc115242 2026-02-22T20:10:28.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'test -n "$image_id"' 2026-02-22T20:10:28.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -n 16dbbc115242 2026-02-22T20:10:28.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2026-02-22T20:10:28.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Flagged image for resync from primary 2026-02-22T20:10:28.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2026-02-22T20:10:28.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_instance=cluster1 2026-02-22T20:10:28.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:28.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:28.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance=cluster1 2026-02-22T20:10:28.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loop_instance 2026-02-22T20:10:28.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:28.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:10:28.389 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:28.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:10:28.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:28.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2026-02-22T20:10:28.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:10:28.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:28.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:10:28.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:28.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T20:10:28.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:10:28.393 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:28.394 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:28.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:28.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:10:28.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:28.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:28.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:28.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:28.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:10:28.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:10:28.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:28.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:28.395 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:28.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2026-02-22T20:10:28.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:28.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test deleted 16dbbc115242 2026-02-22T20:10:28.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:28.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:28.465 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T20:10:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=16dbbc115242 2026-02-22T20:10:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 16dbbc115242 2026-02-22T20:10:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:28.466 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:10:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test deleted 16dbbc115242 2026-02-22T20:10:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:10:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=16dbbc115242 2026-02-22T20:10:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:28.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:28.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:28.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-22T20:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id= 2026-02-22T20:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T20:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2026-02-22T20:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_instance=cluster1 2026-02-22T20:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance=cluster1 2026-02-22T20:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loop_instance 2026-02-22T20:10:28.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:28.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:10:28.595 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:28.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:10:28.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:28.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:28.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:10:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:10:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:28.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:28.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:28.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:28.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2026-02-22T20:10:28.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:28.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test present 2026-02-22T20:10:28.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:28.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:28.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:28.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=present 2026-02-22T20:10:28.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:10:28.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:28.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:28.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:28.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:28.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:28.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:28.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:28.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:28.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id=16db141e49d7 2026-02-22T20:10:28.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:28.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:10:28.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test present 16db141e49d7 2026-02-22T20:10:28.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:28.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:28.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:28.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:10:28.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=16db141e49d7 2026-02-22T20:10:28.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:28.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:28.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:28.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:28.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:28.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:28.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:28.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=16db141e49d7 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 16db141e49d7 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 16db141e49d7 -o 16db141e49d7 = 16db141e49d7 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = present 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:28.840 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:28.841 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:29.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:10:29.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:29.842 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:29.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:29.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:10:29.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:10:29.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:10:29.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:10:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:10:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:10:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:10:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:10:29.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:29.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:10:29.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:29.850 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:10:29.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:29.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:10:29.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:10:29.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:29.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:29.851 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:29.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:29.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:29.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:29.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:29.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:29.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:29.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:29.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:29.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:29.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:29.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:29.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:10:29.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:29.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:29.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:29.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:10:29.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:29.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:29.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:29.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:29.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:29.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:29.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:29.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:10:29.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:10:29.856 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:29.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:29.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:29.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:29.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:10:29.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:10:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:10:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:10:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:10:29.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:10:29.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:10:29.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:29.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:30.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:10:30.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:30.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:10:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:10:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 516307d5-169a-48a7-bdf2-62efacc5d454 2026-02-22T20:10:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:10:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790997,"replay_state":"idle"} 2026-02-22T20:10:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:10:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:10:28 2026-02-22T20:10:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:10:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:10:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:10:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:10:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:10:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:10:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:10:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:10:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790997,"replay_state":"idle"} 2026-02-22T20:10:30.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:10:30.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:30.972 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:10:30.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:30.973 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:10:30.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:10:30.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:10:30.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:10:30.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:28' 2026-02-22T20:10:30.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:10:28' 2026-02-22T20:10:30.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:30.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:30.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:30.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:30.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:10:28' 2026-02-22T20:10:30.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:30.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:30.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:10:31.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING158415851mirror.0trial01719.2.3-689-gf90f4eaetrueOK58455858mirror.1trial01719.2.3-689-gf90f4eaefalseOK58525864mirror.2trial01719.2.3-689-gf90f4eaefalseOK58575869mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest516307d5-169a-48a7-bdf2-62efacc5d454up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790997,"replay_state":"idle"}58415851mirror.0trial0172026-02-22 20:10:28cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:10:31.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:10:31.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:10:31.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:28' 2026-02-22T20:10:31.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:10:31.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:10:31.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:31.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:10:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test 2026-02-22T20:10:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:10:31.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.export 2026-02-22T20:10:31.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.export 2026-02-22T20:10:31.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:31.028 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T20:10:31.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.export 2026-02-22T20:10:31.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.export 2026-02-22T20:10:31.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:31.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:31.031 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:31.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T20:10:31.155 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:31.159 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:31.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:31.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:31.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:31.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: request image resync while daemon is offline' 2026-02-22T20:10:31.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:10:31.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:10:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:10:31 TEST: request image resync while daemon is offline 2026-02-22T20:10:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:10:31 TEST: request image resync while daemon is offline 2026-02-22T20:10:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirrors cluster1 2026-02-22T20:10:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:31.407 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:10:31.408 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:10:31.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:31.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:0 '' 2026-02-22T20:10:31.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:10:31.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:10:31.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:31.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:10:31.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:0 2026-02-22T20:10:31.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:0 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:10:31.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:10:31.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:10:31.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:10:31.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:10:31.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=51620 2026-02-22T20:10:31.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 51620 ']' 2026-02-22T20:10:31.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 51620 2026-02-22T20:10:31.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:31.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:10:31.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:31.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51620 '$2 == pid {print; exit 1}' 2026-02-22T20:10:31.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 51620 1.2 0.0 775128 51232 ? Ssl 20:09 0:00 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:10:31.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:31.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:32.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:32.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51620 '$2 == pid {print; exit 1}' 2026-02-22T20:10:32.456 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:10:32.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:32.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51620 '$2 == pid {print; exit 1}' 2026-02-22T20:10:32.474 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:32.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:10:32.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:32.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:32.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:32.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:32.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:10:32.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:32.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:32.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:32.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:32.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:10:32.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:10:32.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:32.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:32.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:32.489 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:10:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:10:32.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:32.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:32.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:32.500 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:32.501 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:32.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:0 2026-02-22T20:10:32.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:10:32.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:32.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:32.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:32.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:32.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:32.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:32.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:32.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:32.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:32.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:32.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:32.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:32.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:10:32.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:10:32.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:10:32.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:32.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:1 '' 2026-02-22T20:10:32.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:10:32.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:10:32.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:32.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:10:32.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:1 2026-02-22T20:10:32.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:1 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:10:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:10:32.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:10:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:10:32.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:10:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=51648 2026-02-22T20:10:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 51648 ']' 2026-02-22T20:10:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 51648 2026-02-22T20:10:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:32.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:10:32.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:32.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51648 '$2 == pid {print; exit 1}' 2026-02-22T20:10:32.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 51648 0.2 0.0 699316 36732 ? Ssl 20:09 0:00 rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:10:32.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:32.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:33.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:33.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51648 '$2 == pid {print; exit 1}' 2026-02-22T20:10:33.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:10:33.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:33.555 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51648 '$2 == pid {print; exit 1}' 2026-02-22T20:10:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:10:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:10:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:10:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:10:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:10:33.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:33.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:33.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:10:33.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:10:33.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:10:33.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:10:33.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:10:33.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:10:33.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:10:33.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:10:33.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:10:33.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-22T20:10:33.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:10:33.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:10:33.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:33.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:10:33.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:10:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:10:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:10:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:10:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:10:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:10:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:10:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:10:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:10:33.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:10:33.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:10:33.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:1 2026-02-22T20:10:33.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:10:33.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:33.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:10:33.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:10:33.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:33.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:33.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:33.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:10:33.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:10:33.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:33.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:33.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:10:33.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:10:33.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:10:33.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:10:33.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:10:33.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:33.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:2 '' 2026-02-22T20:10:33.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:10:33.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:10:33.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:33.610 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:10:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:2 2026-02-22T20:10:33.611 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:2 2026-02-22T20:10:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:10:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:10:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:10:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:10:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:10:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:10:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:10:33.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:10:33.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:10:33.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:10:33.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:10:33.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:10:33.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=51676 2026-02-22T20:10:33.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 51676 ']' 2026-02-22T20:10:33.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 51676 2026-02-22T20:10:33.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:33.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:10:33.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:33.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51676 '$2 == pid {print; exit 1}' 2026-02-22T20:10:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 51676 0.2 0.0 701368 37440 ? Ssl 20:09 0:00 rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:10:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51676 '$2 == pid {print; exit 1}' 2026-02-22T20:10:34.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:10:34.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:34.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51676 '$2 == pid {print; exit 1}' 2026-02-22T20:10:34.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:10:34.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:10:34.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:34.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:34.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:10:34.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:10:34.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:10:34.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:34.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:34.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:10:34.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:10:34.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:10:34.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:10:34.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:10:34.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:10:34.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:10:34.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:10:34.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:10:34.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-22T20:10:34.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:10:34.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:10:34.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:34.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:10:34.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:10:34.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:10:34.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:34.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:34.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:10:34.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:10:34.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:10:34.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:10:34.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:10:34.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:10:34.680 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:10:34.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:10:34.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:10:34.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:2 2026-02-22T20:10:34.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:10:34.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:34.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:10:34.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:10:34.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:34.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:34.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:34.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:10:34.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:10:34.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:34.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:34.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:10:34.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:10:34.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:10:34.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:10:34.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:10:34.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:34.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:3 '' 2026-02-22T20:10:34.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:10:34.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:10:34.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:34.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:10:34.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:3 2026-02-22T20:10:34.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:3 2026-02-22T20:10:34.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:10:34.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:10:34.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:10:34.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:10:34.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:10:34.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=51736 2026-02-22T20:10:34.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 51736 ']' 2026-02-22T20:10:34.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 51736 2026-02-22T20:10:34.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:34.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:10:34.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:34.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51736 '$2 == pid {print; exit 1}' 2026-02-22T20:10:34.735 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 51736 0.2 0.0 701368 37780 ? Ssl 20:09 0:00 rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:10:34.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:34.736 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:35.737 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51736 '$2 == pid {print; exit 1}' 2026-02-22T20:10:35.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:10:35.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:10:35.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=51736 '$2 == pid {print; exit 1}' 2026-02-22T20:10:35.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:10:35.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:10:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:10:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:10:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:10:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:10:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:10:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:10:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:10:35.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:10:35.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:10:35.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:10:35.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:10:35.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:10:35.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-22T20:10:35.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:10:35.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:10:35.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:35.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:10:35.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:10:35.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:10:35.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:35.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:35.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:10:35.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:10:35.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:10:35.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:10:35.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:10:35.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:10:35.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:10:35.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:10:35.789 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:10:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:3 2026-02-22T20:10:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:10:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:10:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:10:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:10:35.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:10:35.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:35.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:35.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:10:35.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:10:35.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:10:35.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:10:35.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:10:35.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ request_resync_image cluster1 mirror test image_id 2026-02-22T20:10:35.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:35.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:35.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:35.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id_var_name=image_id 2026-02-22T20:10:35.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:35.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:35.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:35.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:35.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:35.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:35.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'image_id='\''16db141e49d7'\''' 2026-02-22T20:10:35.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ image_id=16db141e49d7 2026-02-22T20:10:35.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'test -n "$image_id"' 2026-02-22T20:10:35.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -n 16db141e49d7 2026-02-22T20:10:35.833 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2026-02-22T20:10:35.863 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Flagged image for resync from primary 2026-02-22T20:10:35.867 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirrors cluster1 2026-02-22T20:10:35.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:35.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:10:35.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:0 2026-02-22T20:10:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:10:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:10:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T20:10:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:10:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:10:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:10:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T20:10:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:35.870 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:10:35.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:35.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T20:10:35.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:35.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:35.871 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:10:35.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:35.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:1 2026-02-22T20:10:35.888 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:10:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:10:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:10:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T20:10:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:10:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:10:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:10:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T20:10:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:10:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:10:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T20:10:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:10:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:10:35.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:35.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:2 2026-02-22T20:10:35.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:10:35.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:10:35.916 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:10:35.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T20:10:35.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:10:35.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:10:35.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:10:35.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T20:10:35.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:10:35.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:10:35.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:35.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T20:10:35.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:10:35.917 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:35.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:10:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:3 2026-02-22T20:10:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:10:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:10:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:35.936 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:10:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test deleted 16db141e49d7 2026-02-22T20:10:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=deleted 2026-02-22T20:10:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=16db141e49d7 2026-02-22T20:10:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 16db141e49d7 2026-02-22T20:10:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:35.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:10:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test deleted 16db141e49d7 2026-02-22T20:10:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:36.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:10:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=16db141e49d7 2026-02-22T20:10:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:36.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=16db141e49d7 2026-02-22T20:10:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 16db141e49d7 2026-02-22T20:10:36.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 16db141e49d7 -o 16db141e49d7 = 16db141e49d7 2026-02-22T20:10:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:10:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = present 2026-02-22T20:10:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:36.083 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:37.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test deleted 16db141e49d7 2026-02-22T20:10:37.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:37.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:37.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:37.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:10:37.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=16db141e49d7 2026-02-22T20:10:37.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:37.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:37.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:37.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:37.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:37.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=16db141e49d7 2026-02-22T20:10:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 16db141e49d7 2026-02-22T20:10:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 16db141e49d7 -o 16db141e49d7 = 16db141e49d7 2026-02-22T20:10:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:10:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = present 2026-02-22T20:10:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:37.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:10:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test deleted 16db141e49d7 2026-02-22T20:10:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:10:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=16db141e49d7 2026-02-22T20:10:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:39.118 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:39.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:39.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:39.119 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=16db141e49d7 2026-02-22T20:10:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 16db141e49d7 2026-02-22T20:10:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 16db141e49d7 -o 16db141e49d7 = 16db141e49d7 2026-02-22T20:10:39.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:10:39.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = present 2026-02-22T20:10:39.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:39.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 4 2026-02-22T20:10:43.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test deleted 16db141e49d7 2026-02-22T20:10:43.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:43.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:43.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:43.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=deleted 2026-02-22T20:10:43.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=16db141e49d7 2026-02-22T20:10:43.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:43.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:43.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:43.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:43.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:43.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:43.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:43.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:43.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=178682aebabb 2026-02-22T20:10:43.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 178682aebabb 2026-02-22T20:10:43.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 16db141e49d7 -o 16db141e49d7 = 178682aebabb 2026-02-22T20:10:43.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test deleted = deleted 2026-02-22T20:10:43.189 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:43.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_present cluster1 mirror test present 2026-02-22T20:10:43.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:43.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:43.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:43.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=present 2026-02-22T20:10:43.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id= 2026-02-22T20:10:43.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:43.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:10:43.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:43.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:43.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:43.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:43.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:43.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:43.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image_id=178682aebabb 2026-02-22T20:10:43.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:10:43.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:10:43.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_present cluster1 mirror test present 178682aebabb 2026-02-22T20:10:43.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:43.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:43.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:43.322 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=present 2026-02-22T20:10:43.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id=178682aebabb 2026-02-22T20:10:43.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=deleted 2026-02-22T20:10:43.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_image_id 2026-02-22T20:10:43.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror test 2026-02-22T20:10:43.323 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:43.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:10:43.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test 2026-02-22T20:10:43.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-22T20:10:43.324 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:10:43.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_image_id=178682aebabb 2026-02-22T20:10:43.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 178682aebabb 2026-02-22T20:10:43.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -z 178682aebabb -o 178682aebabb = 178682aebabb 2026-02-22T20:10:43.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=present 2026-02-22T20:10:43.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test present = present 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:43.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:43.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:44.353 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-22T20:10:44.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:44.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:44.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:44.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:10:44.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:10:44.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:10:44.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-22T20:10:44.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:10:44.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:10:44.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:44.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:44.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:10:44.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:10:44.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:44.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:10:44.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:44.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:10:44.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:44.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-22T20:10:44.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:10:44.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:44.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:44.359 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:44.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:44.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:44.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:44.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:44.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:44.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:44.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:44.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:44.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:44.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:44.360 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:10:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:10:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:44.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:44.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:44.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:44.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:10:44.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:10:44.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-22T20:10:44.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:10:44.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:10:44.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:10:44.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:10:44.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:10:44.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:10:44.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:10:44.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:44.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test 2026-02-22T20:10:44.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:44.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:44.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:44.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:10:44.432 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-22T20:10:44.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:44.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:44.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:44.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:44.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:44.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:44.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:44.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:45.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-22T20:10:45.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:45.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:45.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test: 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 516307d5-169a-48a7-bdf2-62efacc5d454 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790997,"replay_state":"idle"} 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:10:40 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:10:45.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:10:45.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:10:45.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:10:45.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790997,"replay_state":"idle"} 2026-02-22T20:10:45.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:10:45.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:45.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:10:45.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:45.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:10:45.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:10:45.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:10:45.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:10:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:40' 2026-02-22T20:10:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-22 20:10:40' 2026-02-22T20:10:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:45.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:45.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:45.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:45.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:10:40' 2026-02-22T20:10:45.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:45.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:45.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:10:45.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING160156022mirror.0trial01719.2.3-689-gf90f4eaetrueOK60196030mirror.1trial01719.2.3-689-gf90f4eaefalseOK60266034mirror.2trial01719.2.3-689-gf90f4eaefalseOK60316037mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest516307d5-169a-48a7-bdf2-62efacc5d454up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771790997,"replay_state":"idle"}60156022mirror.0trial0172026-02-22 20:10:40cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:10:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:10:45.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-22T20:10:45.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:40' 2026-02-22T20:10:45.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-22T20:10:45.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:10:45.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:45.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:10:45.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test 2026-02-22T20:10:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:10:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test.export 2026-02-22T20:10:45.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test.export 2026-02-22T20:10:45.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:45.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T20:10:45.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test.export 2026-02-22T20:10:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test.export 2026-02-22T20:10:45.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:10:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:45.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:45.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export 2026-02-22T20:10:45.666 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:45.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:45.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-22T20:10:45.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:45.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test.export 2026-02-22T20:10:45.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:45.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test 2026-02-22T20:10:45.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:45.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:45.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:45.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:10:45.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:10:45.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test 2026-02-22T20:10:45.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:45.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:45.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test 2026-02-22T20:10:45.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2026-02-22T20:10:45.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test 2026-02-22T20:10:46.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:10:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:10:46.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: split-brain' 2026-02-22T20:10:46.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:10:46.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:10:46.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:10:46 TEST: split-brain 2026-02-22T20:10:46.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:10:46 TEST: split-brain 2026-02-22T20:10:46.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ image=split-brain 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror split-brain snapshot 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror split-brain 2026-02-22T20:10:46.225 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:10:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:10:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:10:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/split-brain 2026-02-22T20:10:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:10:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:46.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:10:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror split-brain snapshot 2026-02-22T20:10:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:10:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:10:46.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/split-brain snapshot 2026-02-22T20:10:47.185 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:10:47.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/split-brain 2026-02-22T20:10:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'split-brain': 2026-02-22T20:10:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 128 MiB in 32 objects 2026-02-22T20:10:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 1809490e7e2a 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.1809490e7e2a 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:10:46 2026 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:10:46 2026 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:10:46 2026 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 1809490e7e2a 2026-02-22T20:10:47.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:10:47.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:10:47.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: e42f7f0c-9468-4813-983f-7c4cc2ae95de 2026-02-22T20:10:47.217 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:10:47.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror split-brain 2026-02-22T20:10:47.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:47.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:47.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:47.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:10:47.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+replaying 2026-02-22T20:10:47.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:47.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:47.219 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying '' '' 2026-02-22T20:10:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:48.221 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:48.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-22T20:10:48.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image split-brain: (2) No such file or directory 2026-02-22T20:10:48.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status= 2026-02-22T20:10:48.246 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:10:48.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:10:48.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:48.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:10:50.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying '' '' 2026-02-22T20:10:50.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:50.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:50.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-22T20:10:50.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='split-brain: 2026-02-22T20:10:50.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: e42f7f0c-9468-4813-983f-7c4cc2ae95de 2026-02-22T20:10:50.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:10:50.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791047,"replay_state":"idle"} 2026-02-22T20:10:50.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:10:50.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:10:49 2026-02-22T20:10:50.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:10:50.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:10:50.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:10:50.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:10:50.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:10:50.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:10:50.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:10:50.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:10:50.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791047,"replay_state":"idle"} 2026-02-22T20:10:50.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:10:50.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:50.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:10:50.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:50.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:10:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:10:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:10:50.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:10:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:49' 2026-02-22T20:10:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+replaying '2026-02-22 20:10:49' 2026-02-22T20:10:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:50.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:50.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:50.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:10:50.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:10:49' 2026-02-22T20:10:50.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:10:50.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:10:50.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:10:50.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING160156022mirror.0trial01719.2.3-689-gf90f4eaetrueOK60196030mirror.1trial01719.2.3-689-gf90f4eaefalseOK60266034mirror.2trial01719.2.3-689-gf90f4eaefalseOK60316037mirror.3trial01719.2.3-689-gf90f4eaefalseOKsplit-braine42f7f0c-9468-4813-983f-7c4cc2ae95deup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791047,"replay_state":"idle"}60156022mirror.0trial0172026-02-22 20:10:49cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:10:50.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:10:50.334 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2026-02-22T20:10:50.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:49' 2026-02-22T20:10:50.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2026-02-22T20:10:50.341 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:10:50.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:10:50.342 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:10:50.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:10:50.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:50.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:50.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror split-brain --force 2026-02-22T20:10:50.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:50.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:50.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:50.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force=--force 2026-02-22T20:10:50.343 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/split-brain --force 2026-02-22T20:10:58.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:10:58.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror split-brain 2026-02-22T20:10:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror split-brain stopped 2026-02-22T20:10:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:10:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:10:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:58.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:10:59.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror split-brain stopped 2026-02-22T20:10:59.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:59.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:59.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:59.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:10:59.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:10:59.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:10:59.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/split-brain 2026-02-22T20:10:59.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:10:59.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:10:59.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:59.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:10:59.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:10:59.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:10:59.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:59.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:10:59.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:10:59.601 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:10:59.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:10:59.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/split-brain 2026-02-22T20:10:59.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:10:59.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:10:59.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:10:59.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:10:59.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:10:59.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:10:59.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:10:59.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:10:59.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:10:59.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:59.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:10:59.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:10:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:10:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:10:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:10:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:10:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:10:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:10:59.606 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:10:59.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:59.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:59.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:10:59.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/split-brain 2026-02-22T20:10:59.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:10:59.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:10:59.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:10:59.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:10:59.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:10:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:10:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+stopped 2026-02-22T20:10:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:10:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:10:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:10:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:10:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:10:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:10:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:10:59.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:11:00.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+stopped '' '' 2026-02-22T20:11:00.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:00.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:11:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:11:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:00.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-22T20:11:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='split-brain: 2026-02-22T20:11:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: e42f7f0c-9468-4813-983f-7c4cc2ae95de 2026-02-22T20:11:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:11:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: orphan (force promoting) 2026-02-22T20:11:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:11:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:10:55 2026-02-22T20:11:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:11:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:11:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:11:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:11:00.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22T20:11:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:11:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 200 .mirror.primary.e42f7f0c-9468-4813-983f-7c4cc2ae95de.9f1585d8-ac5c-4c10-acc9-fd29b4d05c48 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:11:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:11:00.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:11:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:11:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: orphan (force promoting) 2026-02-22T20:11:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:11:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:11:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:11:00.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:11:00.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:11:00.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:11:00.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:11:00.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:11:00.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:55' 2026-02-22T20:11:00.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+stopped '2026-02-22 20:10:55' 2026-02-22T20:11:00.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:00.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:00.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:00.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:11:00.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:10:55' 2026-02-22T20:11:00.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:00.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:00.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:11:00.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING160156022mirror.0trial01719.2.3-689-gf90f4eaetrueOK60196030mirror.1trial01719.2.3-689-gf90f4eaefalseOK60266034mirror.2trial01719.2.3-689-gf90f4eaefalseOK60316037mirror.3trial01719.2.3-689-gf90f4eaefalseOKsplit-braine42f7f0c-9468-4813-983f-7c4cc2ae95deup+stoppedorphan (force promoting)60156022mirror.0trial0172026-02-22 20:10:55cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:11:00.759 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:11:00.760 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2026-02-22T20:11:00.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:10:55' 2026-02-22T20:11:00.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2026-02-22T20:11:00.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:11:00.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:11:00.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:11:00.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:11:00.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:00.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:00.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster1 mirror split-brain 10 2026-02-22T20:11:00.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:00.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:00.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:00.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=10 2026-02-22T20:11:00.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:11:00.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:00.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:11:00.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 bench mirror/split-brain --io-type write --io-size 4096 --io-threads 1 --io-total 40960 --io-pattern rand 2026-02-22T20:11:00.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 40960 pattern random 2026-02-22T20:11:00.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:11:00.814 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 10 ops/sec: 2500.07 bytes/sec: 9.8 MiB/s 2026-02-22T20:11:00.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster1 mirror split-brain 2026-02-22T20:11:00.823 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:00.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:00.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:00.824 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/split-brain 2026-02-22T20:11:01.594 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:11:01.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+error split-brain 2026-02-22T20:11:01.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:01.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:01.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:01.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+error 2026-02-22T20:11:01.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern=split-brain 2026-02-22T20:11:01.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:01.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:01.600 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:11:02.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+error split-brain '' 2026-02-22T20:11:02.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:02.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:02.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:02.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+error 2026-02-22T20:11:02.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern=split-brain 2026-02-22T20:11:02.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:02.602 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:02.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:02.603 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-22T20:11:02.635 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='split-brain: 2026-02-22T20:11:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: e42f7f0c-9468-4813-983f-7c4cc2ae95de 2026-02-22T20:11:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:11:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:11:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:11:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:11:00 2026-02-22T20:11:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:11:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:11:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:11:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:11:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:11:02.636 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+error' 2026-02-22T20:11:02.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:11:02.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:02.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:11:04.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+error split-brain '' 2026-02-22T20:11:04.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:04.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:04.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:04.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+error 2026-02-22T20:11:04.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern=split-brain 2026-02-22T20:11:04.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:04.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:04.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:04.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='split-brain: 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: e42f7f0c-9468-4813-983f-7c4cc2ae95de 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:11:00 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:11:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+error' 2026-02-22T20:11:04.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:11:04.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:04.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 4 2026-02-22T20:11:08.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+error split-brain '' 2026-02-22T20:11:08.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:08.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:08.683 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:08.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+error 2026-02-22T20:11:08.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern=split-brain 2026-02-22T20:11:08.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:08.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:08.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:08.684 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-22T20:11:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='split-brain: 2026-02-22T20:11:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: e42f7f0c-9468-4813-983f-7c4cc2ae95de 2026-02-22T20:11:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+error 2026-02-22T20:11:08.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: split-brain 2026-02-22T20:11:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:11:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:11:05 2026-02-22T20:11:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:11:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:11:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:11:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:11:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:11:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+error' 2026-02-22T20:11:08.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+error 2026-02-22T20:11:08.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*split-brain' 2026-02-22T20:11:08.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: split-brain 2026-02-22T20:11:08.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:11:08.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+error 2026-02-22T20:11:08.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:11:08.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+error 2026-02-22T20:11:08.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:11:08.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:11:08.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:11:08.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:11:08.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:11:05' 2026-02-22T20:11:08.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+error '2026-02-22 20:11:05' 2026-02-22T20:11:08.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:08.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:08.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:08.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+error 2026-02-22T20:11:08.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:11:05' 2026-02-22T20:11:08.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:08.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:08.727 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:11:08.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING160156022mirror.0trial01719.2.3-689-gf90f4eaetrueOK60196030mirror.1trial01719.2.3-689-gf90f4eaefalseOK60266034mirror.2trial01719.2.3-689-gf90f4eaefalseOK60316037mirror.3trial01719.2.3-689-gf90f4eaefalseOKsplit-braine42f7f0c-9468-4813-983f-7c4cc2ae95deup+errorsplit-brain60156022mirror.0trial0172026-02-22 20:11:05cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:11:08.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:11:08.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2026-02-22T20:11:08.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:11:05' 2026-02-22T20:11:08.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2026-02-22T20:11:08.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+error 2026-02-22T20:11:08.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+error 2026-02-22T20:11:08.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+error 2026-02-22T20:11:08.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+error 2026-02-22T20:11:08.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:08.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ request_resync_image cluster1 mirror split-brain image_id 2026-02-22T20:11:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image_id_var_name=image_id 2026-02-22T20:11:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_image_id cluster1 mirror split-brain 2026-02-22T20:11:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=split-brain 2026-02-22T20:11:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster=cluster1 info mirror/split-brain 2026-02-22T20:11:08.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-22T20:11:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'image_id='\''1786b5e19bce'\''' 2026-02-22T20:11:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ image_id=1786b5e19bce 2026-02-22T20:11:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval 'test -n "$image_id"' 2026-02-22T20:11:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -n 1786b5e19bce 2026-02-22T20:11:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/split-brain 2026-02-22T20:11:08.839 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Flagged image for resync from primary 2026-02-22T20:11:08.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror split-brain 2026-02-22T20:11:08.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:11:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+replaying 2026-02-22T20:11:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:11:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:11:08.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:08.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:08.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:11:09.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying '' '' 2026-02-22T20:11:09.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:09.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:09.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:09.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:11:09.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:11:09.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:09.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:09.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:09.845 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-22T20:11:09.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='split-brain: 2026-02-22T20:11:09.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: e42f7f0c-9468-4813-983f-7c4cc2ae95de 2026-02-22T20:11:09.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+error 2026-02-22T20:11:09.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: split-brain 2026-02-22T20:11:09.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:11:09.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:11:05 2026-02-22T20:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:11:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:11:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:09.879 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:11:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying '' '' 2026-02-22T20:11:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:11:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:11:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:11.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:11.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-22T20:11:11.905 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image split-brain: (2) No such file or directory 2026-02-22T20:11:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status= 2026-02-22T20:11:11.908 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:11:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:11:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 4 2026-02-22T20:11:15.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying '' '' 2026-02-22T20:11:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:11:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:11:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-22T20:11:15.937 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd: error opening image split-brain: (2) No such file or directory 2026-02-22T20:11:15.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status= 2026-02-22T20:11:15.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:11:15.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 1 2026-02-22T20:11:15.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:15.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 8 2026-02-22T20:11:23.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying '' '' 2026-02-22T20:11:23.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:23.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:23.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:23.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:11:23.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:11:23.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:23.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:23.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:23.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='split-brain: 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: e42f7f0c-9468-4813-983f-7c4cc2ae95de 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791047,"replay_state":"idle"} 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:11:15 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: down+unknown 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: status not found 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: ' 2026-02-22T20:11:23.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:11:23.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:11:23.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:11:23.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791047,"replay_state":"idle"} 2026-02-22T20:11:23.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:11:23.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:11:23.982 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:11:23.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:11:23.983 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:11:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:11:23.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:11:23.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:11:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:11:15' 2026-02-22T20:11:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+replaying '2026-02-22 20:11:15' 2026-02-22T20:11:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:11:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:11:15' 2026-02-22T20:11:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:23.987 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING160156022mirror.0trial01719.2.3-689-gf90f4eaetrueOK60196030mirror.1trial01719.2.3-689-gf90f4eaefalseOK60266034mirror.2trial01719.2.3-689-gf90f4eaefalseOK60316037mirror.3trial01719.2.3-689-gf90f4eaefalseOKsplit-braine42f7f0c-9468-4813-983f-7c4cc2ae95deup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791047,"replay_state":"idle"}60156022mirror.0trial0172026-02-22 20:11:15cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1down+unknownstatus not found' 2026-02-22T20:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2026-02-22T20:11:24.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:11:15' 2026-02-22T20:11:24.030 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2026-02-22T20:11:24.033 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:11:24.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:11:24.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:11:24.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:11:24.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:24.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:24.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror split-brain 2026-02-22T20:11:24.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:24.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:24.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:24.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:11:24.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:11:24.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror split-brain 2026-02-22T20:11:24.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:24.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:24.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=split-brain 2026-02-22T20:11:24.039 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/split-brain 2026-02-22T20:11:24.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/split-brain 2026-02-22T20:11:25.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-22T20:11:25.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:25.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2026-02-22T20:11:25.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:11:25.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:11:25.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:11:25 TEST: check if removed 'images'\''' OMAP are removed 2026-02-22T20:11:25.021 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:11:25 TEST: check if removed images' OMAP are removed 2026-02-22T20:11:25.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirrors cluster2 2026-02-22T20:11:25.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:25.022 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:11:25.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:25.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster2:0 2026-02-22T20:11:25.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:0 2026-02-22T20:11:25.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:11:25.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:11:25.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster2:0 2026-02-22T20:11:25.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:11:25.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:11:25.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster2 2026-02-22T20:11:25.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T20:11:25.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-22T20:11:25.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster2 2026-02-22T20:11:25.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:11:25.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T20:11:25.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:11:25.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:25.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:11:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster2:1 2026-02-22T20:11:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:1 2026-02-22T20:11:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:11:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:11:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster2:1 2026-02-22T20:11:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:11:25.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:11:25.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster2 2026-02-22T20:11:25.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T20:11:25.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-22T20:11:25.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster2 2026-02-22T20:11:25.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:11:25.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T20:11:25.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:11:25.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:25.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:11:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster2:2 2026-02-22T20:11:25.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:2 2026-02-22T20:11:25.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:11:25.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-22T20:11:25.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster2:2 2026-02-22T20:11:25.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:11:25.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:11:25.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster2 2026-02-22T20:11:25.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T20:11:25.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-22T20:11:25.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster2 2026-02-22T20:11:25.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:11:25.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T20:11:25.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:11:25.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:25.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster2 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:11:25.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:25.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster2:3 2026-02-22T20:11:25.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:3 2026-02-22T20:11:25.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:11:25.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-22T20:11:25.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster2:3 2026-02-22T20:11:25.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:11:25.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:11:25.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster2 2026-02-22T20:11:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T20:11:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-22T20:11:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster2 2026-02-22T20:11:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:11:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T20:11:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:11:25.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:25.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster2 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:11:25.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_in_omap cluster1 mirror 2026-02-22T20:11:25.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:25.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:25.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2026-02-22T20:11:25.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirroring 2026-02-22T20:11:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=status_global 2026-02-22T20:11:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:11:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:11:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T20:11:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2026-02-22T20:11:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:11:25.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=status_global 2026-02-22T20:11:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-22T20:11:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:11:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-22T20:11:25.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c status_global 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirroring 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=image_ 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:11:25.126 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:11:25.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T20:11:25.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2026-02-22T20:11:25.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:25.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:25.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:11:25.127 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=image_ 2026-02-22T20:11:25.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-22T20:11:25.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:25.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:25.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:11:25.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-22T20:11:25.128 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c image_ 2026-02-22T20:11:25.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T20:11:25.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T20:11:25.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T20:11:25.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:11:25.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:25.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2026-02-22T20:11:25.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:25.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:25.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirror_leader 2026-02-22T20:11:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=image_map 2026-02-22T20:11:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:11:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:11:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T20:11:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-22T20:11:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T20:11:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=image_map 2026-02-22T20:11:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-22T20:11:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c image_map 2026-02-22T20:11:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T20:11:25.153 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-22T20:11:25.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 1 = 0 2026-02-22T20:11:25.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=1 2026-02-22T20:11:25.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T20:11:25.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 -eq 0 ']' 2026-02-22T20:11:25.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:11:25.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:11:26.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T20:11:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-22T20:11:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T20:11:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=image_map 2026-02-22T20:11:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-22T20:11:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c image_map 2026-02-22T20:11:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T20:11:26.178 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_in_omap cluster2 mirror 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirroring 2026-02-22T20:11:26.204 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=status_global 2026-02-22T20:11:26.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:11:26.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:11:26.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T20:11:26.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2026-02-22T20:11:26.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:11:26.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:26.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:11:26.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=status_global 2026-02-22T20:11:26.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-22T20:11:26.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c status_global 2026-02-22T20:11:26.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:11:26.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:26.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:11:26.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-22T20:11:26.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T20:11:26.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T20:11:26.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T20:11:26.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:11:26.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:26.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2026-02-22T20:11:26.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:26.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:26.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirroring 2026-02-22T20:11:26.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=image_ 2026-02-22T20:11:26.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:11:26.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:11:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T20:11:26.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2026-02-22T20:11:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:11:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:11:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=image_ 2026-02-22T20:11:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-22T20:11:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:11:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c image_ 2026-02-22T20:11:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirroring 2026-02-22T20:11:26.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-22T20:11:26.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T20:11:26.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T20:11:26.250 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T20:11:26.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:11:26.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:26.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2026-02-22T20:11:26.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:26.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:26.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local obj_name=rbd_mirror_leader 2026-02-22T20:11:26.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local filter=image_map 2026-02-22T20:11:26.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-22T20:11:26.251 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:11:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T20:11:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2026-02-22T20:11:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:11:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T20:11:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local filter=image_map 2026-02-22T20:11:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2026-02-22T20:11:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:11:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:26.253 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -c image_map 2026-02-22T20:11:26.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local obj_name=rbd_mirror_leader 2026-02-22T20:11:26.254 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2026-02-22T20:11:26.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 0 = 0 2026-02-22T20:11:26.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ error_code=0 2026-02-22T20:11:26.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set -e 2026-02-22T20:11:26.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:11:26.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:26.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z '' ']' 2026-02-22T20:11:26.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: no blocklists' 2026-02-22T20:11:26.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:11:26.278 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:11:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:11:26 TEST: no blocklists 2026-02-22T20:11:26.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:11:26 TEST: no blocklists 2026-02-22T20:11:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS='--id admin' 2026-02-22T20:11:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster1 osd blocklist ls 2026-02-22T20:11:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -q 'listed 0 entries' 2026-02-22T20:11:26.486 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS='--id admin' 2026-02-22T20:11:26.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster2 osd blocklist ls 2026-02-22T20:11:26.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -q 'listed 0 entries' 2026-02-22T20:11:26.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:11:26.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z '' ']' 2026-02-22T20:11:26.698 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: test partially synced demote snapshot sync after daemon restart' 2026-02-22T20:11:26.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:11:26.699 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:11:26 TEST: test partially synced demote snapshot sync after daemon restart 2026-02-22T20:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:11:26 TEST: test partially synced demote snapshot sync after daemon restart 2026-02-22T20:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image=test_demote_image 2026-02-22T20:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test_demote_image snapshot 10G 2026-02-22T20:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test_demote_image 10G 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 10G ']' 2026-02-22T20:11:26.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=10G 2026-02-22T20:11:26.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:11:26.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 10G --image-feature layering,exclusive-lock,journaling mirror/test_demote_image 2026-02-22T20:11:26.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:11:26.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:26.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:11:26.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:26.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test_demote_image snapshot 2026-02-22T20:11:26.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:26.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:26.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:26.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:11:26.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test_demote_image snapshot 2026-02-22T20:11:26.984 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:11:26.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test_demote_image 2026-02-22T20:11:27.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_demote_image': 2026-02-22T20:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 10 GiB in 2560 objects 2026-02-22T20:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 184f32e16745 2026-02-22T20:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.184f32e16745 2026-02-22T20:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:11:26 2026 2026-02-22T20:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:11:26 2026 2026-02-22T20:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:11:26 2026 2026-02-22T20:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 184f32e16745 2026-02-22T20:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: 3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e 2026-02-22T20:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:11:27.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_demote_image 100 2026-02-22T20:11:27.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:27.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:27.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:27.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:11:27.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:11:27.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:27.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:11:27.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_demote_image --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:11:27.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:11:27.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:11:27.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.17 bytes/sec: 24 MiB/s 2026-02-22T20:11:27.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_demote_image 2026-02-22T20:11:27.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:27.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:27.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:27.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_demote_image stopped 2026-02-22T20:11:27.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:27.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:11:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:11:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:11:28.084 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_demote_image stopped 2026-02-22T20:11:28.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:28.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:28.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:28.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:11:28.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:11:28.085 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:11:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_demote_image 2026-02-22T20:11:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:11:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:11:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:11:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:11:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:11:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:11:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:11:28.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:11:28.087 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_demote_image 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:11:28.088 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:11:28.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:11:28.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:11:28.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:11:28.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:11:28.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:11:28.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-22T20:11:28.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:11:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:11:28.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:28.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_demote_image 2026-02-22T20:11:28.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:11:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:11:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:11:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:11:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:11:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:11:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:11:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:11:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:11:28.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:11:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:11:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:11:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:11:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:11:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:11:28.167 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:11:28.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-22T20:11:28.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:11:28.234 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:11:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:2 rbd mirror status mirror/test_demote_image 2026-02-22T20:11:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:2 2026-02-22T20:11:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:11:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-22T20:11:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:11:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:11:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:2 cluster2 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:2 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:11:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-22T20:11:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:2 2026-02-22T20:11:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:11:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:11:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:11:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:11:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-22T20:11:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:11:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:11:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:11:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:11:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-22T20:11:28.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:11:28.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:11:28.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:11:28.248 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:11:28.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-22T20:11:28.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:11:28.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:11:28.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:28.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:3 rbd mirror status mirror/test_demote_image 2026-02-22T20:11:28.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:3 2026-02-22T20:11:28.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:11:28.301 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:3 cluster2 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:3 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:3 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:11:28.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:11:28.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-22T20:11:28.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:11:28.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:11:28.311 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:11:28.312 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:11:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_demote_image'] 2026-02-22T20:11:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:11:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:11:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:28.362 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T20:11:29.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:11:29.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:11:29.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:29.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_demote_image 2026-02-22T20:11:29.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:11:29.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:11:29.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:11:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:11:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:11:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:11:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:11:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:11:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:11:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:11:29.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:11:29.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:11:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:11:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:11:29.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:11:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_demote_image 2026-02-22T20:11:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_demote_image started 2026-02-22T20:11:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:11:29.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:11:29.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:29.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:11:30.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_demote_image started 2026-02-22T20:11:30.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:30.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:30.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:30.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:11:30.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:11:30.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:11:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_demote_image 2026-02-22T20:11:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:11:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:11:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:11:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:11:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:11:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:11:30.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:11:30.441 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:11:30.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:30.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_demote_image 2026-02-22T20:11:30.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:11:30.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:11:30.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:30.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:11:30.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:11:30.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:11:30.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:11:30.445 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:11:30.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:11:30.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:11:30.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:11:30.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:11:30.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:11:30.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:11:30.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:11:30.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:11:30.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:11:30.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:11:30.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:11:30.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:11:30.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:11:30.455 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:11:30.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:11:30.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_demote_image 2026-02-22T20:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:30.515 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test_demote_image 2026-02-22T20:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:30.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test_demote_image 2026-02-22T20:11:30.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 192 2026-02-22T20:11:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test_demote_image 2026-02-22T20:11:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:11:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:30.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_demote_image.status 2026-02-22T20:11:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_demote_image.status 2026-02-22T20:11:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:11:30.995 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:11:30.995 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_demote_image.status 2026-02-22T20:11:30.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_demote_image.status 2026-02-22T20:11:30.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:11:30.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:30.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:11:30.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:30.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:30.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:30.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:11:30.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:11:30.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:11:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:11:31.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=192 2026-02-22T20:11:31.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:11:31.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:11:31.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:11:31.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:31.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:31.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:31.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:31.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:31.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:11:31.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:11:31.261 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:11:31.263 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=191 2026-02-22T20:11:31.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 192 = 191 2026-02-22T20:11:31.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:11:31.264 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:11:31.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:31.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:31.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:31.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:31.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:31.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:11:31.665 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:11:31.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:11:31.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:11:31.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:11:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:11:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:11:31.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:11:31.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 192 = '' 2026-02-22T20:11:31.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:11:31.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.8 2026-02-22T20:11:32.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:32.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:32.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:32.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:32.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:32.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:11:32.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:11:32.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:11:32.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=192 2026-02-22T20:11:32.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 192 = 192 2026-02-22T20:11:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_demote_image up+replaying 2026-02-22T20:11:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:11:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:11:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:32.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:11:33.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+replaying '' '' 2026-02-22T20:11:33.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:33.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:33.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:33.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:11:33.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:11:33.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:33.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:33.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:33.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-22T20:11:33.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:11:33.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e 2026-02-22T20:11:33.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:11:33.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791086,"replay_state":"idle"} 2026-02-22T20:11:33.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:11:33.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:11:29 2026-02-22T20:11:33.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:11:33.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:11:33.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:11:33.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:11:33.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:11:30' 2026-02-22T20:11:33.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:11:33.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:11:33.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:11:33.583 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791086,"replay_state":"idle"} 2026-02-22T20:11:33.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:11:33.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:11:33.584 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:11:33.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:11:33.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:11:33.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:11:33.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:11:33.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:11:33.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:11:29' 2026-02-22T20:11:33.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_demote_image up+replaying '2026-02-22 20:11:29' 2026-02-22T20:11:33.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:33.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:33.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:33.589 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:11:33.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:11:29' 2026-02-22T20:11:33.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:33.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:33.590 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:11:33.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK160156022mirror.0trial01719.2.3-689-gf90f4eaetrueOK60196030mirror.1trial01719.2.3-689-gf90f4eaefalseOK60266034mirror.2trial01719.2.3-689-gf90f4eaefalseOK60316037mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_demote_image3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791086,"replay_state":"idle"}60156022mirror.0trial0172026-02-22 20:11:29cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:11:30' 2026-02-22T20:11:33.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:11:33.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:11:33.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:11:29' 2026-02-22T20:11:33.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:11:33.637 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:11:33.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:11:33.638 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:11:33.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:11:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_demote_image up+stopped 2026-02-22T20:11:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:33.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:11:33.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:11:33.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:33.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:11:33.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:11:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_demote_image up+stopped '' '' 2026-02-22T20:11:34.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:11:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:11:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:11:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:34.642 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_demote_image 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:11:30 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791086,"replay_state":"idle"} 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:11:29 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 191 .mirror.primary.3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e.f811bc83-4827-4a5a-8e25-3b1fbd02c061 (peer_uuids:[]) 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 192 .mirror.primary.3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e.1010bfbb-7a53-49ea-94fb-ee2d3c2e2d9d (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:11:34.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:11:34.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:11:34.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:11:34.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:11:34.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:11:34.675 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:11:34.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:11:34.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:11:34.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:11:34.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:11:34.678 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:11:34.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:11:34.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:11:30' 2026-02-22T20:11:34.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_demote_image up+stopped '2026-02-22 20:11:30' 2026-02-22T20:11:34.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:34.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:34.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:34.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:11:34.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:11:30' 2026-02-22T20:11:34.681 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:11:34.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:11:34.682 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:11:34.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK161696176mirror.0trial01719.2.3-689-gf90f4eaetrueOK61736185mirror.1trial01719.2.3-689-gf90f4eaefalseOK61806190mirror.2trial01719.2.3-689-gf90f4eaefalseOK61876193mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_demote_image3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2eup+stoppedlocal image is primary61696176mirror.0trial0172026-02-22 20:11:30cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791086,"replay_state":"idle"}2026-02-22 20:11:29' 2026-02-22T20:11:34.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:11:34.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:11:34.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:11:30' 2026-02-22T20:11:34.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:11:34.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:11:34.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:11:34.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:11:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:11:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_demote_image 2560 4194304 2026-02-22T20:11:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=2560 2026-02-22T20:11:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=4194304 2026-02-22T20:11:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 4194304 2026-02-22T20:11:34.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_demote_image --io-type write --io-size 4194304 --io-threads 1 --io-total 10737418240 --io-pattern rand 2026-02-22T20:11:34.761 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4194304 io_threads 1 bytes 10737418240 pattern random 2026-02-22T20:11:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:11:35.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 1 138 139.004 556 MiB/s 2026-02-22T20:11:36.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 2 275 137.728 551 MiB/s 2026-02-22T20:11:37.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 3 398 132.827 531 MiB/s 2026-02-22T20:11:38.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 4 532 133.121 532 MiB/s 2026-02-22T20:11:39.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 5 666 132.661 531 MiB/s 2026-02-22T20:11:40.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 6 768 125.601 502 MiB/s 2026-02-22T20:11:41.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 7 905 125.903 504 MiB/s 2026-02-22T20:11:42.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 8 1025 125.203 501 MiB/s 2026-02-22T20:11:43.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 9 1160 125.303 501 MiB/s 2026-02-22T20:11:44.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 10 1292 125.807 503 MiB/s 2026-02-22T20:11:45.762 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 11 1409 128.615 514 MiB/s 2026-02-22T20:11:46.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 12 1530 125.103 500 MiB/s 2026-02-22T20:11:47.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 13 1653 125.906 504 MiB/s 2026-02-22T20:11:48.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 14 1781 124.602 498 MiB/s 2026-02-22T20:11:49.763 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 15 1910 123.702 495 MiB/s 2026-02-22T20:11:50.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 16 2029 123.509 494 MiB/s 2026-02-22T20:11:51.764 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 17 2155 125.103 500 MiB/s 2026-02-22T20:11:52.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 18 2267 122.607 490 MiB/s 2026-02-22T20:11:53.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 19 2393 122.208 489 MiB/s 2026-02-22T20:11:54.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 20 2495 116.63 467 MiB/s 2026-02-22T20:11:55.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 20 ops: 2560 ops/sec: 124.107 bytes/sec: 496 MiB/s 2026-02-22T20:11:55.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ demote_image cluster2 mirror test_demote_image 2026-02-22T20:11:55.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:55.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:55.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:55.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test_demote_image 2026-02-22T20:11:57.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image demoted to non-primary 2026-02-22T20:11:57.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_complete_mirror_snapshot_id cluster2 mirror test_demote_image primary_snap_id 2026-02-22T20:11:57.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:57.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:57.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:57.026 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local -n _snap_id=primary_snap_id 2026-02-22T20:11:57.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:11:57.027 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '(//snapshots/snapshot[namespace/complete='\''true'\'']/id)[last()]' 2026-02-22T20:11:57.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ _snap_id=193 2026-02-22T20:11:57.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_non_primary_snap_present cluster1 mirror test_demote_image 193 2026-02-22T20:11:57.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:57.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:57.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:57.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snap_id=193 2026-02-22T20:11:57.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:11:57.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-22T20:11:57.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.1 2026-02-22T20:11:57.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:11:57.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot/namespace[primary_snap_id='\''193'\'']' 2026-02-22T20:11:57.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:11:57.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirrors cluster1 -KILL 2026-02-22T20:11:57.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:57.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig=-KILL 2026-02-22T20:11:57.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:11:57.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:57.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:0 -KILL 2026-02-22T20:11:57.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:11:57.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig=-KILL 2026-02-22T20:11:57.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:57.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:11:57.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:0 2026-02-22T20:11:57.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:0 2026-02-22T20:11:57.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:11:57.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:11:57.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:11:57.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:11:57.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:11:57.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:11:57.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:11:57.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:11:57.196 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:11:57.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:11:57.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:11:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=54118 2026-02-22T20:11:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 54118 ']' 2026-02-22T20:11:57.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill -KILL 54118 2026-02-22T20:11:57.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:11:57.211 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:11:57.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:11:57.212 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=54118 '$2 == pid {print; exit 1}' 2026-02-22T20:11:57.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:11:57.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:11:57.244 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=54118 '$2 == pid {print; exit 1}' 2026-02-22T20:11:57.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:11:57.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:11:57.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:57.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:11:57.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:11:57.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:11:57.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:11:57.276 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:11:57.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:11:57.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:11:57.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:11:57.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:11:57.277 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:11:57.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:11:57.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:11:57.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-22T20:11:57.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:11:57.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:11:57.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:11:57.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:11:57.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:11:57.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:11:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:0 2026-02-22T20:11:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:11:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:11:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:11:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:11:57.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:11:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:11:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:11:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:11:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:11:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:11:57.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:11:57.313 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:11:57.314 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:1 -KILL 2026-02-22T20:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig=-KILL 2026-02-22T20:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:1 2026-02-22T20:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:1 2026-02-22T20:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:11:57.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:11:57.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:11:57.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:11:57.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:11:57.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:11:57.327 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:11:57.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=54146 2026-02-22T20:11:57.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 54146 ']' 2026-02-22T20:11:57.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill -KILL 54146 2026-02-22T20:11:57.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:11:57.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:11:57.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:11:57.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=54146 '$2 == pid {print; exit 1}' 2026-02-22T20:11:57.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:11:57.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:11:57.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=54146 '$2 == pid {print; exit 1}' 2026-02-22T20:11:57.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:11:57.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:11:57.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:11:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:11:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:11:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:11:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:11:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:11:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:11:57.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:11:57.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:11:57.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:11:57.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:11:57.397 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:11:57.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-22T20:11:57.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:11:57.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:11:57.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:11:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:11:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:11:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:11:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:11:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:11:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:11:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:11:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:11:57.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:11:57.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:11:57.411 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:11:57.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:1 2026-02-22T20:11:57.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:11:57.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:11:57.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:11:57.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:11:57.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:11:57.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:11:57.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:11:57.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:11:57.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:11:57.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:11:57.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:11:57.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:11:57.423 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:11:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:57.424 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:2 -KILL 2026-02-22T20:11:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:11:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig=-KILL 2026-02-22T20:11:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:57.425 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:11:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:2 2026-02-22T20:11:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:2 2026-02-22T20:11:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:11:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:11:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:11:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:11:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:11:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:11:57.427 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:11:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:11:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:11:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:11:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:11:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:11:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:11:57.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:11:57.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:11:57.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=54190 2026-02-22T20:11:57.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 54190 ']' 2026-02-22T20:11:57.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill -KILL 54190 2026-02-22T20:11:57.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:11:57.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:11:57.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:11:57.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=54190 '$2 == pid {print; exit 1}' 2026-02-22T20:11:57.467 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:11:57.468 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:11:57.468 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=54190 '$2 == pid {print; exit 1}' 2026-02-22T20:11:57.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:11:57.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:11:57.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:57.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:11:57.498 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:11:57.499 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:11:57.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:11:57.508 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:11:57.509 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-22T20:11:57.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:11:57.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:11:57.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:11:57.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:11:57.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:11:57.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.510 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:11:57.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:11:57.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:11:57.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:11:57.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:11:57.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:11:57.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:11:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:11:57.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:11:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:2 2026-02-22T20:11:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:11:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:11:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:11:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:11:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:11:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:11:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:11:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:11:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:11:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:11:57.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:11:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:11:57.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:11:57.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:57.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:3 -KILL 2026-02-22T20:11:57.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:11:57.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig=-KILL 2026-02-22T20:11:57.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:57.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:11:57.534 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:3 2026-02-22T20:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:3 2026-02-22T20:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:11:57.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:11:57.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:11:57.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:11:57.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:11:57.536 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:11:57.544 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:11:57.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:11:57.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=54234 2026-02-22T20:11:57.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 54234 ']' 2026-02-22T20:11:57.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill -KILL 54234 2026-02-22T20:11:57.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:11:57.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:11:57.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:11:57.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=54234 '$2 == pid {print; exit 1}' 2026-02-22T20:11:57.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:11:57.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:11:57.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=54234 '$2 == pid {print; exit 1}' 2026-02-22T20:11:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:11:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:11:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:11:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:11:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:11:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.604 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:11:57.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:11:57.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:11:57.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:11:57.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:11:57.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:11:57.605 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:11:57.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:11:57.616 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:11:57.617 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:11:57.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:11:57.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:11:57.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:11:57.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:3 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:11:57.630 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:11:57.631 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:11:57.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:11:57.640 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:11:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_snaps_json cluster1 mirror test_demote_image 2026-02-22T20:11:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:11:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:11:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_demote_image 2026-02-22T20:11:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 snap ls mirror/test_demote_image --all --format json 2026-02-22T20:11:57.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ SNAPS='[{"id":204,"name":".mirror.non_primary.3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e.fbc90542-0824-4a77-a472-e293c46d9efb","size":10737418240,"protected":"false","timestamp":"Sun Feb 22 20:11:31 2026","namespace":{"type":"mirror","state":"non-primary","mirror_peer_uuids":[],"complete":true,"primary_mirror_uuid":"5ef5496a-89f6-490a-8cc8-3f2dd61214a1","primary_snap_id":192,"last_copied_object_number":2560}},{"id":205,"name":".mirror.non_primary.3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e.547bf772-c822-4571-80d5-976b5d01f39f","size":10737418240,"protected":"false","timestamp":"Sun Feb 22 20:11:56 2026","namespace":{"type":"mirror","state":"demoted","mirror_peer_uuids":["c42be08c-54e0-47b4-8faa-b0889455bae6"],"complete":false,"primary_mirror_uuid":"5ef5496a-89f6-490a-8cc8-3f2dd61214a1","primary_snap_id":193,"last_copied_object_number":189}}]' 2026-02-22T20:11:57.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ jq -e '.[-1].namespace["type"] == "mirror" and .[-1].namespace["state"] == "demoted" and .[-1].namespace["complete"] == false' 2026-02-22T20:11:57.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:mirrordemotedc42be08c-54e0-47b4-8faa-b0889455bae6false5ef5496a-89f6-490a-8cc8-3f2dd61214a1193181true 2026-02-22T20:11:57.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirrors cluster1 2026-02-22T20:11:57.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:57.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:0 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:1 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T20:11:57.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:11:57.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:57.712 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:11:57.729 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:2 2026-02-22T20:11:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:11:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:11:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:11:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T20:11:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:11:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:11:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:11:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T20:11:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:11:57.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:11:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T20:11:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:11:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:11:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:11:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:3 2026-02-22T20:11:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:11:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:11:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:11:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T20:11:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:11:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:11:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:11:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T20:11:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:11:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:11:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:11:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T20:11:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:11:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:11:57.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:11:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test_demote_image 2026-02-22T20:11:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:11:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_demote_image.status 2026-02-22T20:11:57.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_demote_image.status 2026-02-22T20:11:57.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:11:57.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:11:57.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_demote_image.status 2026-02-22T20:11:57.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_demote_image.status 2026-02-22T20:11:57.769 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:11:57.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:57.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:11:57.770 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:11:57.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:57.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:57.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.status 2026-02-22T20:11:57.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:11:57.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:11:57.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:11:57.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=193 2026-02-22T20:11:57.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:11:57.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:11:57.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:11:58.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:58.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:58.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:58.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:58.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:58.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:11:58.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:11:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:11:58.034 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:11:58.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:11:58.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:11:58.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:11:58.036 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:11:58.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = '' 2026-02-22T20:11:58.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:11:58.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:11:58.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:58.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:58.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:58.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:58.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:58.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:11:58.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:11:58.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:11:58.470 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:11:58.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:11:58.471 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = '' 2026-02-22T20:11:58.472 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:11:58.473 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.8 2026-02-22T20:11:59.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:59.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:11:59.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:11:59.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:11:59.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:11:59.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:11:59.275 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:11:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:11:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:11:59.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:11:59.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:11:59.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:11:59.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:11:59.308 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = '' 2026-02-22T20:11:59.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:11:59.309 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1.6 2026-02-22T20:12:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:12:00.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:12:00.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:12:00.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:00.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:12:00.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:12:00.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:12:00.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:12:00.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:12:00.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:12:00.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:12:00.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:12:00.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = '' 2026-02-22T20:12:00.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:12:00.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:12:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:12:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:12:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:12:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:12:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:12:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:12:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:12:02.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:12:02.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:12:02.979 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:12:02.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:12:02.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = '' 2026-02-22T20:12:02.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:12:02.980 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:12:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:12:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:12:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:12:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:12:04.981 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:12:05.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:12:05.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:12:05.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:12:05.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:12:05.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:12:05.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:12:05.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = '' 2026-02-22T20:12:05.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:12:05.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 4 2026-02-22T20:12:09.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:09.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:12:09.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:12:09.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:12:09.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:09.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:12:09.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:12:09.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:12:09.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:12:09.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:12:09.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:12:09.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:12:09.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:12:09.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = '' 2026-02-22T20:12:09.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:12:09.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 4 2026-02-22T20:12:13.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:13.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:12:13.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:12:13.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:12:13.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:13.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:12:13.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:12:13.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:12:13.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:12:13.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:12:13.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:12:13.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:12:13.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:12:13.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = '' 2026-02-22T20:12:13.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:12:13.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 8 2026-02-22T20:12:21.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:12:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:12:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:12:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:12:21.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:12:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:12:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:12:21.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:12:21.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:12:21.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:12:21.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:12:21.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = '' 2026-02-22T20:12:21.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:12:21.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 8 2026-02-22T20:12:29.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:29.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:12:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:12:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:12:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:12:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:12:29.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:12:29.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:12:29.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:12:29.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:12:29.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:12:29.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:12:29.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = '' 2026-02-22T20:12:29.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:12:29.367 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 16 2026-02-22T20:12:45.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:45.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:12:45.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:12:45.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:12:45.368 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:12:45.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:12:45.369 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:12:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:12:45.400 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:12:45.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:12:45.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:12:45.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:12:45.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:12:45.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = '' 2026-02-22T20:12:45.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:12:45.403 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 16 2026-02-22T20:13:01.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:13:01.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:01.405 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:01.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:01.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.status 2026-02-22T20:13:01.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_demote_image --format xml 2026-02-22T20:13:01.406 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:13:01.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:13:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=193 2026-02-22T20:13:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 193 = 193 2026-02-22T20:13:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_demote_image up+unknown 2026-02-22T20:13:01.442 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:13:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:13:02.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_demote_image up+unknown '' '' 2026-02-22T20:13:02.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:02.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:02.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:02.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:13:02.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:02.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:02.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:02.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:02.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_demote_image 2026-02-22T20:13:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:13:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e 2026-02-22T20:13:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:13:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:13:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:13:02.475 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:13:00 2026-02-22T20:13:02.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:13:02.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:13:02.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:13:02.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image demoted 2026-02-22T20:13:02.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:12:59' 2026-02-22T20:13:02.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:13:02.476 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:13:02.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:13:02.477 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:13:02.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:13:02.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:13:02.478 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:13:02.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:13:02.479 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:13:02.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:13:02.480 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:13:02.481 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:13:02.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:00' 2026-02-22T20:13:02.483 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_demote_image up+unknown '2026-02-22 20:13:00' 2026-02-22T20:13:02.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:02.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:02.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:02.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:13:02.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:13:00' 2026-02-22T20:13:02.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:02.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:02.484 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:13:02.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING161696176mirror.0trial01719.2.3-689-gf90f4eaetrueOK61736185mirror.1trial01719.2.3-689-gf90f4eaefalseOK61806190mirror.2trial01719.2.3-689-gf90f4eaefalseOK61876193mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_demote_image3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2eup+unknownremote image is not primary61696176mirror.0trial0172026-02-22 20:13:00cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+unknownremote image demoted2026-02-22 20:12:59' 2026-02-22T20:13:02.520 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:13:02.521 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:13:02.523 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:00' 2026-02-22T20:13:02.524 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:13:02.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:13:02.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:13:02.527 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:13:02.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:13:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_demote_image up+unknown 2026-02-22T20:13:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:13:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:13:03.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+unknown '' '' 2026-02-22T20:13:03.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:03.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:03.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:03.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:13:03.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:03.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:03.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:03.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:03.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-22T20:13:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:13:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e 2026-02-22T20:13:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:13:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:13:03.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:13:03.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:13:02 2026-02-22T20:13:03.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:13:03.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:13:03.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+unknown 2026-02-22T20:13:03.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: remote image is not primary 2026-02-22T20:13:03.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:13:00' 2026-02-22T20:13:03.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+unknown' 2026-02-22T20:13:03.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+unknown 2026-02-22T20:13:03.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:13:03.567 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: remote image is not primary 2026-02-22T20:13:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:13:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:13:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:13:03.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:13:03.569 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:13:03.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:13:03.570 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:13:03.571 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:02' 2026-02-22T20:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_demote_image up+unknown '2026-02-22 20:13:02' 2026-02-22T20:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+unknown 2026-02-22T20:13:03.573 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:13:02' 2026-02-22T20:13:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:03.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:13:03.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='WARNINGOKWARNING161866197mirror.0trial01719.2.3-689-gf90f4eaefalseOK61906201mirror.1trial01719.2.3-689-gf90f4eaetrueOK61946206mirror.2trial01719.2.3-689-gf90f4eaefalseOK62036209mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_demote_image3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2eup+unknownremote image is not primary61866197mirror.0trial0172026-02-22 20:13:02cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+unknownremote image is not primary2026-02-22 20:13:00' 2026-02-22T20:13:03.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:13:03.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:13:03.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:02' 2026-02-22T20:13:03.615 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:13:03.618 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+unknown 2026-02-22T20:13:03.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+unknown 2026-02-22T20:13:03.619 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+unknown 2026-02-22T20:13:03.620 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+unknown 2026-02-22T20:13:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ compare_images mirror test_demote_image 2026-02-22T20:13:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:03.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local ret=0 2026-02-22T20:13:03.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_demote_image.export 2026-02-22T20:13:03.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_demote_image.export 2026-02-22T20:13:03.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:13:03.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local rmt_export=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.export 2026-02-22T20:13:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_demote_image.export 2026-02-22T20:13:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_demote_image.export 2026-02-22T20:13:03.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:13:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local loc_export=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.export 2026-02-22T20:13:03.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.export 2026-02-22T20:13:03.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 export mirror/test_demote_image /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.export 2026-02-22T20:13:11.317 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:13:11.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 export mirror/test_demote_image /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.export 2026-02-22T20:13:20.130 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-22T20:13:20.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cmp /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.export 2026-02-22T20:13:24.872 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_demote_image.export /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_demote_image.export 2026-02-22T20:13:26.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:26.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test_demote_image 2026-02-22T20:13:26.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:26.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:26.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:26.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force= 2026-02-22T20:13:26.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_demote_image 2026-02-22T20:13:27.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:13:27.714 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster2 mirror test_demote_image 2026-02-22T20:13:27.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:27.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:27.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:27.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_demote_image started 2026-02-22T20:13:27.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:27.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:27.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:27.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:13:27.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:13:27.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:27.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:13:28.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_demote_image started 2026-02-22T20:13:28.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:28.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:28.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:28.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:13:28.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:13:28.717 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:13:28.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_demote_image 2026-02-22T20:13:28.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:13:28.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:13:28.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:28.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:13:28.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:13:28.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:13:28.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:28.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:13:28.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:13:28.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:13:28.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:13:28.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_demote_image 2026-02-22T20:13:28.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:13:28.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:28.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:13:28.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:13:28.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:13:28.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:13:28.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:13:28.722 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:13:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:13:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:13:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:13:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:13:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:13:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:13:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:13:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:13:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:13:28.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:28.732 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:28.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:28.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:13:28.797 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:13:28.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:13:28.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:13:28.798 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:13:28.799 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = stopped 2026-02-22T20:13:28.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:28.800 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:13:30.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_demote_image started 2026-02-22T20:13:30.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:30.801 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:30.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:30.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:13:30.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:13:30.802 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:13:30.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_demote_image 2026-02-22T20:13:30.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:13:30.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:13:30.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:30.803 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:13:30.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:13:30.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:13:30.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:30.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:13:30.804 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:13:30.805 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:13:30.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:13:30.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_demote_image 2026-02-22T20:13:30.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:13:30.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:30.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:13:30.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:13:30.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:13:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:13:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:13:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:13:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:13:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:13:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:13:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:13:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:13:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:13:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:13:30.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:13:30.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:13:30.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:13:30.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:13:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:30.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_demote_image 2026-02-22T20:13:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:13:30.873 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:13:30.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:13:30.874 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:13:30.875 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:13:30.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:13:30.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:13:30.876 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:30.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_demote_image up+stopped 2026-02-22T20:13:30.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:30.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:30.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:30.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:13:30.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:30.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:30.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:30.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:13:31.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_demote_image up+stopped '' '' 2026-02-22T20:13:31.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:31.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:31.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:31.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:13:31.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:31.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:31.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:31.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:31.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_demote_image 2026-02-22T20:13:31.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:13:31.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e 2026-02-22T20:13:31.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:13:31.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:13:31.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:13:31.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:13:27 2026-02-22T20:13:31.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:13:31.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:13:31.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:13:31.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791207,"replay_state":"idle"} 2026-02-22T20:13:31.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:13:30 2026-02-22T20:13:31.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:13:31.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 206 .mirror.primary.3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e.b196c9ce-48d4-4e5f-9a4a-9f9019f9e72d (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:13:31.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:13:31.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:13:31.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:13:31.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:13:31.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:13:31.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:13:31.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:13:31.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:13:31.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:13:31.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:13:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:13:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:13:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:27' 2026-02-22T20:13:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_demote_image up+stopped '2026-02-22 20:13:27' 2026-02-22T20:13:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:13:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:13:27' 2026-02-22T20:13:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:13:31.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK161866197mirror.0trial01719.2.3-689-gf90f4eaefalseOK61906201mirror.1trial01719.2.3-689-gf90f4eaetrueOK61946206mirror.2trial01719.2.3-689-gf90f4eaefalseOK62036209mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_demote_image3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2eup+stoppedlocal image is primary61866197mirror.0trial0172026-02-22 20:13:27cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791207,"replay_state":"idle"}2026-02-22 20:13:30' 2026-02-22T20:13:31.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:13:31.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:13:31.990 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:27' 2026-02-22T20:13:31.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:13:31.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:13:31.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:13:31.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:13:31.995 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:13:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_demote_image up+replaying 2026-02-22T20:13:31.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:31.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:31.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:31.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:13:31.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:31.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:31.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:31.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:13:32.997 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_demote_image up+replaying '' '' 2026-02-22T20:13:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:13:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:32.998 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_demote_image 2026-02-22T20:13:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_demote_image: 2026-02-22T20:13:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: 3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2e 2026-02-22T20:13:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:13:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791207,"replay_state":"idle"} 2026-02-22T20:13:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:13:33.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:13:30 2026-02-22T20:13:33.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:13:33.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:13:33.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:13:33.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:13:33.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:13:32' 2026-02-22T20:13:33.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:13:33.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:13:33.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:13:33.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791207,"replay_state":"idle"} 2026-02-22T20:13:33.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:13:33.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:13:33.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:13:33.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:13:33.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:13:33.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:13:33.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:13:33.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:13:33.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:30' 2026-02-22T20:13:33.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_demote_image up+replaying '2026-02-22 20:13:30' 2026-02-22T20:13:33.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:33.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:33.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:33.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:13:33.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:13:30' 2026-02-22T20:13:33.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:33.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:33.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:13:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK161696176mirror.0trial01719.2.3-689-gf90f4eaetrueOK61736185mirror.1trial01719.2.3-689-gf90f4eaefalseOK61806190mirror.2trial01719.2.3-689-gf90f4eaefalseOK61876193mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_demote_image3fdd58f6-a7a0-45a3-abe7-3a3ff14b8f2eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791207,"replay_state":"idle"}61696176mirror.0trial0172026-02-22 20:13:30cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:13:32' 2026-02-22T20:13:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:13:33.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/last_update' 2026-02-22T20:13:33.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:30' 2026-02-22T20:13:33.103 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_demote_image'\'']/state' 2026-02-22T20:13:33.105 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:13:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:13:33.106 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:13:33.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:13:33.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:33.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:33.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster1 mirror test_demote_image 2026-02-22T20:13:33.108 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:33.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:33.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:33.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:13:33.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:13:33.109 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster1 mirror test_demote_image 2026-02-22T20:13:33.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:33.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:33.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_demote_image 2026-02-22T20:13:33.110 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_demote_image 2026-02-22T20:13:33.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 rm mirror/test_demote_image 2026-02-22T20:13:35.487 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:13:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ testlog 'TEST: force promote with a user snapshot' 2026-02-22T20:13:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ tee -a /tmp/tmp.bhaDYZBKee/rbd-mirror.test.log 2026-02-22T20:13:35.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ date '+%F %T' 2026-02-22T20:13:35.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo 2026-02-22 20:13:35 TEST: force promote with a user snapshot 2026-02-22T20:13:35.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:2026-02-22 20:13:35 TEST: force promote with a user snapshot 2026-02-22T20:13:35.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ force_promote_image=test_force_promote_user 2026-02-22T20:13:35.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image_and_enable_mirror cluster2 mirror test_force_promote_user snapshot 10G 2026-02-22T20:13:35.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:35.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:13:35.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:35.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:13:35.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:35.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:13:35.495 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n snapshot ']' 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_image cluster2 mirror test_force_promote_user 10G 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=128 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 10G ']' 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=10G 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ shift 2026-02-22T20:13:35.496 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 create --size 10G --image-feature layering,exclusive-lock,journaling mirror/test_force_promote_user 2026-02-22T20:13:35.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ is_pool_mirror_mode_image mirror 2026-02-22T20:13:35.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:35.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' image = image ']' 2026-02-22T20:13:35.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:35.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ enable_mirror cluster2 mirror test_force_promote_user snapshot 2026-02-22T20:13:35.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:35.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:35.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:35.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local mode=snapshot 2026-02-22T20:13:35.525 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test_force_promote_user snapshot 2026-02-22T20:13:36.041 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Mirroring enabled 2026-02-22T20:13:36.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 info mirror/test_force_promote_user 2026-02-22T20:13:36.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_force_promote_user': 2026-02-22T20:13:36.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 10 GiB in 2560 objects 2026-02-22T20:13:36.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 1 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 189262d179c 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.189262d179c 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:13:35 2026 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:13:35 2026 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:13:35 2026 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 189262d179c 2026-02-22T20:13:36.074 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:13:36.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:13:36.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: fff73af1-224f-4b01-8b7f-7beeb4565034 2026-02-22T20:13:36.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: true 2026-02-22T20:13:36.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_force_promote_user 100 2026-02-22T20:13:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:13:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:13:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:13:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:13:36.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote_user --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:13:36.120 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:13:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:13:36.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.16 bytes/sec: 24 MiB/s 2026-02-22T20:13:36.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote_user 2026-02-22T20:13:36.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:36.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:36.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:36.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote_user stopped 2026-02-22T20:13:36.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:36.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:36.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:36.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:13:36.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:13:36.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:36.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:13:37.146 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_force_promote_user stopped 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:37.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:13:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:13:37.148 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:13:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:13:37.149 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:13:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:13:37.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:13:37.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:13:37.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:13:37.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:13:37.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:37.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:37.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:37.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:37.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:13:37.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:13:37.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:13:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:13:37.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:37.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:13:37.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:13:37.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:13:37.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:13:37.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:13:37.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:13:37.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:13:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:13:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:13:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:13:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:37.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:13:37.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:13:37.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:2 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:2 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:13:37.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:2 cluster2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:13:37.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-22T20:13:37.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:13:37.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:13:37.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:13:37.304 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:37.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:13:37.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:13:37.361 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:3 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:3 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:3 cluster2 2026-02-22T20:13:37.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:3 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:3 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:13:37.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-22T20:13:37.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:13:37.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:13:37.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:13:37.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:37.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:13:37.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:13:37.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:13:37.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:37.435 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T20:13:38.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:13:38.436 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:13:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:13:38.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:13:38.438 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:13:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:13:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:13:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:13:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:13:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:13:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:13:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:13:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:13:38.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:13:38.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:38.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:38.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:13:38.454 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:38.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:13:38.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:13:38.526 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:13:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:13:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:13:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:38.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:13:38.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:13:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:13:38.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:13:38.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:13:38.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:13:38.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:13:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:13:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:13:38.546 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:38.621 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:13:38.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:13:38.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:13:38.622 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:13:38.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:13:38.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:38.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_started cluster1 mirror test_force_promote_user 2026-02-22T20:13:38.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:38.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:38.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:38.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote_user started 2026-02-22T20:13:38.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:38.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:38.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:38.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=started 2026-02-22T20:13:38.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:13:38.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:38.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:13:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_force_promote_user started 2026-02-22T20:13:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=started 2026-02-22T20:13:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:13:39.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:13:39.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:39.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:13:39.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:13:39.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:39.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:13:39.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:13:39.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:13:39.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:39.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:13:39.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:13:39.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:13:39.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:13:39.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:39.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:13:39.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:39.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:13:39.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:13:39.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:13:39.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:13:39.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:13:39.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:13:39.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:13:39.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:13:39.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:13:39.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:13:39.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:13:39.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:13:39.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:13:39.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:13:39.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:13:39.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:13:39.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:13:39.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:13:39.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:13:39.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:13:39.644 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:39.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:13:39.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:13:39.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:13:39.705 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:13:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:13:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:13:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:13:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:13:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:13:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:13:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:13:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:13:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:13:39.706 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:13:39.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:13:39.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:13:39.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:13:39.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:13:39.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:13:39.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:13:39.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:13:39.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:13:39.708 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:13:39.715 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:13:39.716 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:13:39.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:13:39.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Replaying"' 2026-02-22T20:13:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Replaying"' 2026-02-22T20:13:39.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:13:39.783 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: "state": "Replaying" 2026-02-22T20:13:39.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ current_state=started 2026-02-22T20:13:39.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test started = started 2026-02-22T20:13:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_force_promote_user 2026-02-22T20:13:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:13:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:13:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:13:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test_force_promote_user 2026-02-22T20:13:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:39.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:39.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:39.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test_force_promote_user 2026-02-22T20:13:40.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 196 2026-02-22T20:13:40.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test_force_promote_user 2026-02-22T20:13:40.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_cluster=cluster1 2026-02-22T20:13:40.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:40.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:40.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:40.058 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster2-mirror-test_force_promote_user.status 2026-02-22T20:13:40.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster2-mirror-test_force_promote_user.status 2026-02-22T20:13:40.059 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:13:40.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_force_promote_user.status 2026-02-22T20:13:40.060 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ mkfname cluster1-mirror-test_force_promote_user.status 2026-02-22T20:13:40.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo cluster1-mirror-test_force_promote_user.status 2026-02-22T20:13:40.061 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-22T20:13:40.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local local_status_log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote_user.status 2026-02-22T20:13:40.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster2 mirror test_force_promote_user /tmp/tmp.bhaDYZBKee/cluster2-mirror-test_force_promote_user.status 2026-02-22T20:13:40.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:40.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:40.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:40.063 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster2-mirror-test_force_promote_user.status 2026-02-22T20:13:40.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap list --all mirror/test_force_promote_user --format xml 2026-02-22T20:13:40.064 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:13:40.092 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-22T20:13:40.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snapshot_id=196 2026-02-22T20:13:40.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:13:40.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:13:40.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.2 2026-02-22T20:13:40.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_force_promote_user /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote_user.status 2026-02-22T20:13:40.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:40.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:40.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:40.298 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote_user.status 2026-02-22T20:13:40.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_force_promote_user --format xml 2026-02-22T20:13:40.299 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:13:40.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:13:40.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=195 2026-02-22T20:13:40.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 196 = 195 2026-02-22T20:13:40.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:13:40.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.4 2026-02-22T20:13:40.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_force_promote_user /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote_user.status 2026-02-22T20:13:40.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:40.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:40.733 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:40.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote_user.status 2026-02-22T20:13:40.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_force_promote_user --format xml 2026-02-22T20:13:40.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:13:40.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ true 2026-02-22T20:13:40.768 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:13:40.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:-:1.1: Document is empty 2026-02-22T20:13:40.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 2026-02-22T20:13:40.771 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:^ 2026-02-22T20:13:40.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id= 2026-02-22T20:13:40.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 196 = '' 2026-02-22T20:13:40.772 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-22T20:13:40.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0.8 2026-02-22T20:13:41.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ get_newest_mirror_snapshot cluster1 mirror test_force_promote_user /tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote_user.status 2026-02-22T20:13:41.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:41.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:41.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:41.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local log=/tmp/tmp.bhaDYZBKee/cluster1-mirror-test_force_promote_user.status 2026-02-22T20:13:41.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 snap list --all mirror/test_force_promote_user --format xml 2026-02-22T20:13:41.575 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-22T20:13:41.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-22T20:13:41.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local primary_snapshot_id=196 2026-02-22T20:13:41.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test 196 = 196 2026-02-22T20:13:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_force_promote_user 2026-02-22T20:13:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = journal ']' 2026-02-22T20:13:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote_user up+replaying 2026-02-22T20:13:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:41.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:13:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:13:42.612 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote_user up+replaying '' '' 2026-02-22T20:13:42.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:42.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:42.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:42.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:13:42.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:42.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:42.613 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:42.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:42.614 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote_user 2026-02-22T20:13:42.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_force_promote_user: 2026-02-22T20:13:42.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: fff73af1-224f-4b01-8b7f-7beeb4565034 2026-02-22T20:13:42.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:13:42.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791216,"replay_state":"idle"} 2026-02-22T20:13:42.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:13:42.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:13:37 2026-02-22T20:13:42.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:13:42.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:13:42.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:13:42.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:13:42.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:13:40' 2026-02-22T20:13:42.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+replaying' 2026-02-22T20:13:42.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+replaying 2026-02-22T20:13:42.652 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:13:42.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791216,"replay_state":"idle"} 2026-02-22T20:13:42.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:13:42.653 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:13:42.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:13:42.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:13:42.654 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:13:42.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:13:42.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:13:42.656 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:13:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:37' 2026-02-22T20:13:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote_user up+replaying '2026-02-22 20:13:37' 2026-02-22T20:13:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:13:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+replaying 2026-02-22T20:13:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:13:37' 2026-02-22T20:13:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:42.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:42.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:13:42.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK161866197mirror.0trial01719.2.3-689-gf90f4eaefalseOK61906201mirror.1trial01719.2.3-689-gf90f4eaetrueOK61946206mirror.2trial01719.2.3-689-gf90f4eaefalseOK62036209mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_force_promote_userfff73af1-224f-4b01-8b7f-7beeb4565034up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791216,"replay_state":"idle"}61906201mirror.1trial0172026-02-22 20:13:37cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:13:40' 2026-02-22T20:13:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:13:42.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/last_update' 2026-02-22T20:13:42.902 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:37' 2026-02-22T20:13:42.903 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/state' 2026-02-22T20:13:42.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+replaying 2026-02-22T20:13:42.906 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+replaying 2026-02-22T20:13:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+replaying 2026-02-22T20:13:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+replaying 2026-02-22T20:13:42.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:42.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:42.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote_user up+stopped 2026-02-22T20:13:42.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:42.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:42.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:42.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:13:42.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:42.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:42.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:13:42.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:13:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote_user up+stopped '' '' 2026-02-22T20:13:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote_user 2026-02-22T20:13:43.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_force_promote_user: 2026-02-22T20:13:43.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: fff73af1-224f-4b01-8b7f-7beeb4565034 2026-02-22T20:13:43.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:13:43.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:13:43.942 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:13:40 2026-02-22T20:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+replaying 2026-02-22T20:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791216,"replay_state":"idle"} 2026-02-22T20:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:13:37 2026-02-22T20:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 195 .mirror.primary.fff73af1-224f-4b01-8b7f-7beeb4565034.5fa1e601-cba3-4ee0-97d5-1976fe945cec (peer_uuids:[]) 2026-02-22T20:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 196 .mirror.primary.fff73af1-224f-4b01-8b7f-7beeb4565034.40989be1-bd3f-4b5e-848e-8339a6012054 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:13:43.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:13:43.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:13:43.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:13:43.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:13:43.946 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:13:43.947 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:13:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:13:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:13:43.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:40' 2026-02-22T20:13:43.950 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote_user up+stopped '2026-02-22 20:13:40' 2026-02-22T20:13:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:13:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:13:40' 2026-02-22T20:13:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:13:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:13:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:13:43.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK161696176mirror.0trial01719.2.3-689-gf90f4eaetrueOK61736185mirror.1trial01719.2.3-689-gf90f4eaefalseOK61806190mirror.2trial01719.2.3-689-gf90f4eaefalseOK61876193mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_force_promote_userfff73af1-224f-4b01-8b7f-7beeb4565034up+stoppedlocal image is primary61736185mirror.1trial0172026-02-22 20:13:40cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771791216,"replay_state":"idle"}2026-02-22 20:13:37' 2026-02-22T20:13:43.985 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:13:43.986 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/last_update' 2026-02-22T20:13:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:13:40' 2026-02-22T20:13:43.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/state' 2026-02-22T20:13:43.991 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:13:43.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:13:43.992 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:13:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:13:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:13:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_force_promote_user 100 2026-02-22T20:13:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:13:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:13:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:13:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:13:43.994 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote_user --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:13:44.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:13:44.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:13:44.247 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 5000.12 bytes/sec: 20 MiB/s 2026-02-22T20:13:44.255 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ create_snapshot cluster2 mirror test_force_promote_user snap1 2026-02-22T20:13:44.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:44.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:44.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:44.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap=snap1 2026-02-22T20:13:44.256 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 snap create mirror/test_force_promote_user@snap1 2026-02-22T20:13:45.177 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2026-02-22T20:13:45.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_force_promote_user 2560 4194304 2026-02-22T20:13:45.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:13:45.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:13:45.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:13:45.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=2560 2026-02-22T20:13:45.186 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size=4194304 2026-02-22T20:13:45.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n 4194304 2026-02-22T20:13:45.187 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote_user --io-type write --io-size 4194304 --io-threads 1 --io-total 10737418240 --io-pattern rand 2026-02-22T20:13:45.222 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4194304 io_threads 1 bytes 10737418240 pattern random 2026-02-22T20:13:46.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:13:46.233 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 1 126 126.497 506 MiB/s 2026-02-22T20:13:47.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 2 257 129.003 516 MiB/s 2026-02-22T20:13:48.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 3 389 129.314 517 MiB/s 2026-02-22T20:13:49.232 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 4 519 129.873 519 MiB/s 2026-02-22T20:13:50.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 5 643 128.392 514 MiB/s 2026-02-22T20:13:50.254 INFO:tasks.ceph.cluster2.osd.0.trial017.stderr:2026-02-22T20:13:50.257+0000 7f6b45dad640 -1 reset not still connected to 0x555a478f9ee0 2026-02-22T20:13:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 6 770 128.906 516 MiB/s 2026-02-22T20:13:52.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 7 893 127.101 508 MiB/s 2026-02-22T20:13:53.249 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 8 1026 127.301 509 MiB/s 2026-02-22T20:13:54.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 9 1145 125.303 501 MiB/s 2026-02-22T20:13:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 10 1285 128.815 515 MiB/s 2026-02-22T20:13:56.230 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 11 1414 128.7 515 MiB/s 2026-02-22T20:13:57.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 12 1537 128.906 516 MiB/s 2026-02-22T20:13:58.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 13 1669 128.809 515 MiB/s 2026-02-22T20:13:59.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 14 1797 130.403 522 MiB/s 2026-02-22T20:14:00.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 15 1917 126.403 506 MiB/s 2026-02-22T20:14:01.231 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 16 2043 125.803 503 MiB/s 2026-02-22T20:14:02.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 17 2164 125.403 502 MiB/s 2026-02-22T20:14:03.237 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 18 2297 125.704 503 MiB/s 2026-02-22T20:14:04.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 19 2411 122.803 491 MiB/s 2026-02-22T20:14:05.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: 20 2544 125.403 502 MiB/s 2026-02-22T20:14:05.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 20 ops: 2560 ops/sec: 127.012 bytes/sec: 508 MiB/s 2026-02-22T20:14:05.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:14:05.625 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ mirror_image_snapshot cluster2 mirror test_force_promote_user 2026-02-22T20:14:05.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:14:05.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:14:05.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:14:05.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test_force_promote_user 2026-02-22T20:14:07.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Snapshot ID: 198 2026-02-22T20:14:07.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_snap_present cluster1 mirror test_force_promote_user snap1 2026-02-22T20:14:07.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:14:07.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:14:07.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:14:07.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local snap_name=snap1 2026-02-22T20:14:07.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:14:07.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-22T20:14:07.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:14:08.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 info mirror/test_force_promote_user@snap1 2026-02-22T20:14:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:rbd image 'test_force_promote_user': 2026-02-22T20:14:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: size 10 GiB in 2560 objects 2026-02-22T20:14:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: order 22 (4 MiB objects) 2026-02-22T20:14:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: snapshot_count: 3 2026-02-22T20:14:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: id: 1839589bde4b 2026-02-22T20:14:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: block_name_prefix: rbd_data.1839589bde4b 2026-02-22T20:14:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: format: 2 2026-02-22T20:14:08.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: features: layering, exclusive-lock, journaling, non-primary 2026-02-22T20:14:08.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: op_features: 2026-02-22T20:14:08.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: flags: 2026-02-22T20:14:08.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: create_timestamp: Sun Feb 22 20:13:37 2026 2026-02-22T20:14:08.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: access_timestamp: Sun Feb 22 20:13:37 2026 2026-02-22T20:14:08.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: modify_timestamp: Sun Feb 22 20:13:37 2026 2026-02-22T20:14:08.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: protected: False 2026-02-22T20:14:08.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: journal: 1839589bde4b 2026-02-22T20:14:08.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring state: enabled 2026-02-22T20:14:08.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring mode: snapshot 2026-02-22T20:14:08.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring global id: fff73af1-224f-4b01-8b7f-7beeb4565034 2026-02-22T20:14:08.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: mirroring primary: false 2026-02-22T20:14:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:14:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:14:09.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirrors cluster1 -KILL 2026-02-22T20:14:09.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:14:09.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig=-KILL 2026-02-22T20:14:09.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:14:09.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:09.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:0 -KILL 2026-02-22T20:14:09.053 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:14:09.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig=-KILL 2026-02-22T20:14:09.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:14:09.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:14:09.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:0 2026-02-22T20:14:09.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:0 2026-02-22T20:14:09.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:09.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:14:09.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:14:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:14:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:14:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:14:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:14:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:14:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:14:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:14:09.066 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:14:09.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=57357 2026-02-22T20:14:09.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 57357 ']' 2026-02-22T20:14:09.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill -KILL 57357 2026-02-22T20:14:09.068 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:14:09.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:14:09.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:14:09.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=57357 '$2 == pid {print; exit 1}' 2026-02-22T20:14:09.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:14:09.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:14:09.107 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=57357 '$2 == pid {print; exit 1}' 2026-02-22T20:14:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:14:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:14:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:14:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:14:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:14:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:14:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:14:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:14:09.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:14:09.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:14:09.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:09.145 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:14:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:09.158 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:09.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-22T20:14:09.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:14:09.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:14:09.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:14:09.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:09.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:09.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:0 2026-02-22T20:14:09.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:14:09.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:14:09.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:14:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:14:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:14:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:09.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:14:09.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:14:09.188 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:14:09.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:09.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:1 -KILL 2026-02-22T20:14:09.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:14:09.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig=-KILL 2026-02-22T20:14:09.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:14:09.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:14:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:1 2026-02-22T20:14:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:1 2026-02-22T20:14:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:14:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:14:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:14:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:09.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:14:09.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:14:09.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:14:09.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:14:09.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:14:09.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:14:09.205 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:14:09.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=57385 2026-02-22T20:14:09.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 57385 ']' 2026-02-22T20:14:09.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill -KILL 57385 2026-02-22T20:14:09.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:14:09.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:14:09.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:14:09.208 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=57385 '$2 == pid {print; exit 1}' 2026-02-22T20:14:09.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:14:09.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:14:09.243 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=57385 '$2 == pid {print; exit 1}' 2026-02-22T20:14:09.279 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:14:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:14:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:14:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:14:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:14:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:14:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:14:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:14:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:14:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:14:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:14:09.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:09.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-22T20:14:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:14:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:14:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:14:09.293 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:14:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:14:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:14:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:14:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:14:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:14:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:14:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:09.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:09.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:1 2026-02-22T20:14:09.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:14:09.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:14:09.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:14:09.305 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:09.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:14:09.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:09.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:09.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:09.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:14:09.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:09.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:14:09.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:14:09.315 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:14:09.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:09.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:2 -KILL 2026-02-22T20:14:09.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:14:09.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig=-KILL 2026-02-22T20:14:09.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:14:09.316 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:14:09.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:2 2026-02-22T20:14:09.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:2 2026-02-22T20:14:09.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:09.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:14:09.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:14:09.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:14:09.319 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:14:09.328 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:14:09.329 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:14:09.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=57413 2026-02-22T20:14:09.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 57413 ']' 2026-02-22T20:14:09.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill -KILL 57413 2026-02-22T20:14:09.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:14:09.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:14:09.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:14:09.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=57413 '$2 == pid {print; exit 1}' 2026-02-22T20:14:09.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:14:09.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:14:09.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=57413 '$2 == pid {print; exit 1}' 2026-02-22T20:14:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:14:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:14:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:14:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:14:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:14:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:14:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:14:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:14:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:14:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:14:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:09.410 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:14:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:09.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-22T20:14:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:14:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:14:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:14:09.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:14:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:14:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:14:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:14:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:14:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:14:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:09.422 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:14:09.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:09.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:09.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:2 2026-02-22T20:14:09.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:14:09.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:14:09.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:14:09.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:09.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.433 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:14:09.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:09.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:09.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:09.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:14:09.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:09.434 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:14:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:14:09.443 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:14:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:3 -KILL 2026-02-22T20:14:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:14:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig=-KILL 2026-02-22T20:14:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:14:09.444 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:14:09.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:3 2026-02-22T20:14:09.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:3 2026-02-22T20:14:09.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:09.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:14:09.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:14:09.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:14:09.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:09.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:09.446 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:09.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:09.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:14:09.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:14:09.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:14:09.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:14:09.447 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:14:09.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:14:09.457 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:14:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=57489 2026-02-22T20:14:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 57489 ']' 2026-02-22T20:14:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill -KILL 57489 2026-02-22T20:14:09.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:14:09.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:14:09.459 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:14:09.460 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=57489 '$2 == pid {print; exit 1}' 2026-02-22T20:14:09.493 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:14:09.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:14:09.494 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=57489 '$2 == pid {print; exit 1}' 2026-02-22T20:14:09.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:14:09.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:14:09.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:14:09.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.528 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:14:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:09.539 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:09.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-22T20:14:09.541 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:14:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:14:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:14:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:14:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:14:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:14:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:14:09.542 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:14:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:14:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:09.543 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:14:09.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:09.551 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:09.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:3 2026-02-22T20:14:09.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:14:09.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:09.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:14:09.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:14:09.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:09.552 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:09.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:09.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:14:09.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:09.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:09.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:09.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:14:09.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:09.553 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:14:09.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:14:09.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:14:09.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' snapshot = snapshot ']' 2026-02-22T20:14:09.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ get_snaps_json cluster1 mirror test_force_promote_user 2026-02-22T20:14:09.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:14:09.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local pool=mirror 2026-02-22T20:14:09.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local image=test_force_promote_user 2026-02-22T20:14:09.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 snap ls mirror/test_force_promote_user --all --format json 2026-02-22T20:14:09.598 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ SNAPS='[{"id":208,"name":".mirror.non_primary.fff73af1-224f-4b01-8b7f-7beeb4565034.9f02f056-2c94-4917-a77a-377e0b75c348","size":10737418240,"protected":"false","timestamp":"Sun Feb 22 20:13:40 2026","namespace":{"type":"mirror","state":"non-primary","mirror_peer_uuids":[],"complete":true,"primary_mirror_uuid":"5ef5496a-89f6-490a-8cc8-3f2dd61214a1","primary_snap_id":196,"last_copied_object_number":2560}},{"id":209,"name":"snap1","size":10737418240,"protected":"false","timestamp":"Sun Feb 22 20:14:06 2026","namespace":{"type":"user"}},{"id":210,"name":".mirror.non_primary.fff73af1-224f-4b01-8b7f-7beeb4565034.172bc9ce-e418-4283-bc38-729ca1dc86ec","size":10737418240,"protected":"false","timestamp":"Sun Feb 22 20:14:07 2026","namespace":{"type":"mirror","state":"non-primary","mirror_peer_uuids":[],"complete":false,"primary_mirror_uuid":"5ef5496a-89f6-490a-8cc8-3f2dd61214a1","primary_snap_id":198,"last_copied_object_number":599}}]' 2026-02-22T20:14:09.599 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ jq -e '.[-1].namespace["type"] == "mirror" and .[-1].namespace["state"] == "non-primary" and .[-1].namespace["complete"] == false' 2026-02-22T20:14:09.608 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:true 2026-02-22T20:14:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ promote_image cluster1 mirror test_force_promote_user --force 2026-02-22T20:14:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:14:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:14:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:14:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local force=--force 2026-02-22T20:14:09.609 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_force_promote_user --force 2026-02-22T20:14:43.641 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:Image promoted to primary 2026-02-22T20:14:43.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirrors cluster1 2026-02-22T20:14:43.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:14:43.649 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:14:43.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:43.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:0 2026-02-22T20:14:43.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:14:43.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:14:43.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:14:43.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:0 2026-02-22T20:14:43.650 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:14:43.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:14:43.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:14:43.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=0 2026-02-22T20:14:43.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:43.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:14:43.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:43.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=0 2026-02-22T20:14:43.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:14:43.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:14:43.651 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:14:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:1 2026-02-22T20:14:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:14:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:14:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:14:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:1 2026-02-22T20:14:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:14:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:14:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:14:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=1 2026-02-22T20:14:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:14:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=1 2026-02-22T20:14:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:14:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:14:43.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:14:43.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:2 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:2 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=2 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=2 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:14:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:14:43.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:43.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ start_mirror cluster1:3 2026-02-22T20:14:43.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:14:43.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local val=cluster1:3 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster_var_name=cluster 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local instance_var_name=instance 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cluster=cluster1 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ instance=3 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval cluster=cluster1 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ eval instance=3 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:14:43.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:14:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster1 mirror test_force_promote_user 2026-02-22T20:14:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:14:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:14:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:14:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote_user stopped 2026-02-22T20:14:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:14:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:14:43.742 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:14:43.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:14:43.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:14:43.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:14:43.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:14:44.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster1 mirror test_force_promote_user stopped 2026-02-22T20:14:44.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:14:44.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:14:44.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:14:44.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:14:44.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:14:44.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:14:44.743 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:44.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:14:44.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster1 2026-02-22T20:14:44.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:44.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:14:44.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster1 2026-02-22T20:14:44.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:14:44.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:14:44.744 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:14:44.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:14:44.745 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:14:44.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:44.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:44.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:14:44.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:44.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:44.748 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:14:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:14:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:14:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:14:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:44.749 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:44.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:14:44.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:14:44.751 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:14:44.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:44.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:44.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:44.758 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:44.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:44.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:44.808 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:44.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:44.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:14:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:14:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:14:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:14:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:44.811 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:44.812 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:44.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:44.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:44.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:44.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:44.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:14:44.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:14:44.813 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:14:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:44.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:44.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:44.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:44.877 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:44.878 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:44.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:44.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:44.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:14:44.880 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:14:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:14:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:14:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:44.881 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:44.882 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:14:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:14:44.883 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:14:44.890 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:44.891 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:44.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:44.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:44.948 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:14:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:44.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:14:44.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:14:44.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:14:44.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:14:44.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:44.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:44.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:44.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:45.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:45.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:45.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:45.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:14:45.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 1 2026-02-22T20:14:46.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:14:46.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:14:46.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:46.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:46.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:14:46.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:46.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:46.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:14:46.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:14:46.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:46.017 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:14:46.018 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:14:46.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:46.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:46.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:46.029 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:46.089 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:46.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:46.090 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:46.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:46.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:46.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:14:46.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:46.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:46.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:14:46.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:14:46.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:46.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:46.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:46.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:14:46.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:46.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:46.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:46.096 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:14:46.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:46.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:14:46.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:14:46.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:46.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:46.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:14:46.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:14:46.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:46.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:46.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:46.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:46.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:46.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:46.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:46.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:14:46.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:14:46.099 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:14:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:46.104 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:46.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:46.160 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:46.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:14:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:14:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:14:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:46.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:14:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:14:46.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:14:46.174 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:46.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:46.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:46.175 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:46.226 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:46.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:46.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:46.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:14:46.227 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:14:46.228 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:14:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:14:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:14:46.229 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:14:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:46.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:46.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:46.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:46.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:14:46.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 2 2026-02-22T20:14:48.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:14:48.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:14:48.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:48.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:48.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:14:48.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:48.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:48.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:14:48.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:14:48.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:48.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:48.296 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:14:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:14:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:14:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:14:48.297 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:14:48.306 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:48.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:48.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:48.307 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:48.365 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:48.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:48.366 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:48.370 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:14:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:14:48.371 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:14:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:14:48.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:48.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:48.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:48.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:14:48.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:48.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:14:48.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:14:48.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:48.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:48.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:14:48.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:14:48.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:48.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:48.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:48.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:48.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:48.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:48.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:48.375 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:14:48.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:14:48.376 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:14:48.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:48.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:48.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:48.381 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:48.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:48.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:48.437 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:48.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:48.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:48.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:14:48.439 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:14:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:14:48.450 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:48.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:48.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:48.451 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:48.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:48.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:48.507 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:48.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:48.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:48.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:14:48.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:48.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:48.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:14:48.511 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:14:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:14:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:14:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:14:48.512 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:48.513 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:14:48.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:14:48.514 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:14:48.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:48.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:48.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:48.522 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:48.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:48.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:48.579 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:48.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:14:48.580 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 4 2026-02-22T20:14:52.581 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:14:52.582 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:14:52.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:52.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:52.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:14:52.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:52.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:52.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:14:52.585 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:14:52.586 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:14:52.588 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:14:52.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:52.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:52.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:14:52.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:52.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:52.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:52.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:14:52.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:52.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:52.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:14:52.659 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:52.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:52.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:52.720 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:52.723 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:52.725 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:14:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:14:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:14:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:14:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:14:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:14:52.734 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:52.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:52.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:52.782 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:52.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:14:52.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:52.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:14:52.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:14:52.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:14:52.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:14:52.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:14:52.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:14:52.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:14:52.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:14:52.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:14:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:14:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:14:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:14:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:14:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:14:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:14:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:14:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:14:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:14:52.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:14:52.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:52.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:52.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:14:52.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:14:52.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:14:52.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:14:52.852 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:14:52.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:14:52.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 8 2026-02-22T20:15:00.854 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:00.855 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:15:00.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:00.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:00.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:00.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:00.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:00.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:00.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:15:00.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:00.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:00.859 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:15:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:15:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:15:00.860 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:00.868 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:00.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:00.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:00.869 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:00.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:00.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:00.929 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:00.930 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:15:00.931 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:15:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:00.943 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:00.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:00.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:00.944 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:01.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:01.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:01.001 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:01.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:01.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:01.004 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:15:01.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:01.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:01.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:01.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:15:01.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:01.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:01.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:01.005 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:15:01.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:01.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:01.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:01.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:15:01.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:01.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:01.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:15:01.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:01.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:01.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:01.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:15:01.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:01.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:01.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:01.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:01.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:01.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:01.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:01.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:15:01.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:15:01.007 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:01.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:01.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:01.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:01.016 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:01.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:01.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:15:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:01.075 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:15:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:15:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:15:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:15:01.076 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:01.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:15:01.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:15:01.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:01.077 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:15:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:15:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:01.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:01.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:15:01.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:15:01.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:15:01.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:01.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:01.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:01.086 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:01.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:01.136 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 8 2026-02-22T20:15:09.138 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:09.139 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:15:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:15:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:09.142 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:15:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:15:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:15:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:09.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:09.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:09.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:09.161 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:09.209 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:09.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:09.210 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:09.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:09.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:09.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:15:09.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:09.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:09.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:09.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:15:09.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:09.213 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:15:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:15:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:15:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:09.214 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:15:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:15:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:15:09.216 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:09.223 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:09.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:09.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:09.224 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:09.280 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:09.281 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:09.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:09.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:09.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:15:09.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:09.283 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:09.284 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:09.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:15:09.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:09.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:09.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:09.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:15:09.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:09.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:09.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:09.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:09.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:09.285 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:09.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:09.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:15:09.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:15:09.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:09.294 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:09.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:09.295 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:09.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:09.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:09.351 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:15:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:15:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:15:09.355 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:15:09.356 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:15:09.357 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:15:09.363 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:09.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:09.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:09.364 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:09.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:09.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:09.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:09.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:09.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 8 2026-02-22T20:15:17.414 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:17.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:15:17.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:17.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:17.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:17.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:17.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:17.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:17.418 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:15:17.419 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:15:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:15:17.421 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:17.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:17.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:17.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:17.431 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:17.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:17.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:17.490 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:15:17.491 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:1 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:15:17.492 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:17.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:17.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:17.502 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:17.503 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:17.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:17.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:17.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:17.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:17.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:17.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:15:17.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:17.563 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:17.564 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:2 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:17.565 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:15:17.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:15:17.566 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:17.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:17.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:17.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:17.574 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:17.623 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:17.624 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:15:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:17.626 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:17.627 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:3 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:17.628 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:17.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:15:17.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:15:17.629 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:15:17.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:17.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:17.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:17.639 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:17.686 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:17.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:17.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 8 2026-02-22T20:15:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-22T20:15:25.689 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:15:25.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:25.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:25.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:25.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:25.692 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster1:0 2026-02-22T20:15:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1 2026-02-22T20:15:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:15:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster1 2026-02-22T20:15:25.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster1 2026-02-22T20:15:25.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:15:25.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:15:25.695 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:25.703 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:25.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:25.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:25.704 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:25.773 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:15:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:15:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:15:25.775 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:15:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:15:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:15:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote_user 2026-02-22T20:15:25.776 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:15:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote_user stopped 2026-02-22T20:15:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:15:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state=stopped 2026-02-22T20:15:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local s 2026-02-22T20:15:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:15:26.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_image_replay_state cluster2 mirror test_force_promote_user stopped 2026-02-22T20:15:26.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:15:26.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:26.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:26.779 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local test_state=stopped 2026-02-22T20:15:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status_result 2026-02-22T20:15:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local current_state=stopped 2026-02-22T20:15:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ grep -i state 2026-02-22T20:15:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_instance=cluster2 2026-02-22T20:15:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:15:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance=cluster2 2026-02-22T20:15:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local loop_instance 2026-02-22T20:15:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:26.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sleep 0 2026-02-22T20:15:26.784 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' cluster2 '!=' cluster2 ']' 2026-02-22T20:15:26.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ seq 0 3 2026-02-22T20:15:26.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:26.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:26.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:15:26.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:26.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:26.785 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:15:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:15:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:15:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:15:26.786 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:15:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:15:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:15:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:0 2026-02-22T20:15:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:15:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:15:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:15:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:26.787 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:15:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:15:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:15:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:15:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:15:26.788 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:15:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:15:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:15:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:15:26.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:26.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:Invalid command: unused arguments: ['mirror/test_force_promote_user'] 2026-02-22T20:15:26.843 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:rbd mirror status : get status for rbd mirror 2026-02-22T20:15:26.844 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:admin_socket: invalid command 2026-02-22T20:15:26.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:26.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:26.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:15:26.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ shift 2026-02-22T20:15:26.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:26.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:15:26.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:15:26.846 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:26.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:26.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:26.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:15:26.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:15:26.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:15:26.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:26.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:15:26.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:26.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:15:26.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local local_cluster=cluster2:1 2026-02-22T20:15:26.847 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2 2026-02-22T20:15:26.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:26.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:15:26.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:15:26.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=local_cluster 2026-02-22T20:15:26.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:26.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:26.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:26.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:15:26.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval local_cluster=cluster2 2026-02-22T20:15:26.848 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ local_cluster=cluster2 2026-02-22T20:15:26.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:15:26.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:15:26.849 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:15:26.857 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:15:26.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:15:26.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:15:26.858 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote_user 2026-02-22T20:15:26.918 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ return 0 2026-02-22T20:15:26.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status_result=' "state": "Stopped"' 2026-02-22T20:15:26.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo ' "state": "Stopped"' 2026-02-22T20:15:26.919 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep -i Replaying 2026-02-22T20:15:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test stopped = stopped 2026-02-22T20:15:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:15:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote_user up+stopped 2026-02-22T20:15:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:15:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:15:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:15:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:15:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:15:27.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote_user up+stopped '' '' 2026-02-22T20:15:27.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:15:27.922 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:15:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:15:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:15:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:15:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:15:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote_user 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_force_promote_user: 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: fff73af1-224f-4b01-8b7f-7beeb4565034 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.0 on trial017 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:15:23 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster2-DNE 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:15:25 2026-02-22T20:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 212 .mirror.primary.fff73af1-224f-4b01-8b7f-7beeb4565034.1d3b2d32-f0e8-4061-9cc0-394527a30554 (peer_uuids:[c42be08c-54e0-47b4-8faa-b0889455bae6])' 2026-02-22T20:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:15:27.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:15:27.960 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:15:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:15:27.961 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:15:27.962 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.0 on trial017 2026-02-22T20:15:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:15:27.963 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:15:27.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:15:23' 2026-02-22T20:15:27.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote_user up+stopped '2026-02-22 20:15:23' 2026-02-22T20:15:27.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:15:27.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:27.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:27.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:15:27.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:15:23' 2026-02-22T20:15:27.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:15:27.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:15:27.965 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-22T20:15:28.002 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK163026310mirror.0trial01719.2.3-689-gf90f4eaetrueOK63056315mirror.1trial01719.2.3-689-gf90f4eaefalseOK63126322mirror.2trial01719.2.3-689-gf90f4eaefalseOK63196325mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_force_promote_userfff73af1-224f-4b01-8b7f-7beeb4565034up+stoppedlocal image is primary63026310mirror.0trial0172026-02-22 20:15:23cluster2-DNE5ef5496a-89f6-490a-8cc8-3f2dd61214a1up+stoppedlocal image is primary2026-02-22 20:15:25' 2026-02-22T20:15:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:15:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/last_update' 2026-02-22T20:15:28.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:15:23' 2026-02-22T20:15:28.006 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/state' 2026-02-22T20:15:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:15:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:15:28.010 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:15:28.011 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:15:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:15:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:15:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote_user up+stopped 2026-02-22T20:15:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:15:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:15:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:15:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:15:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-22T20:15:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:15:29.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote_user up+stopped '' '' 2026-02-22T20:15:29.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:15:29.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:29.013 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:29.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:15:29.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local description_pattern= 2026-02-22T20:15:29.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local service_pattern= 2026-02-22T20:15:29.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:15:29.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:15:29.014 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote_user 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='test_force_promote_user: 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: global_id: fff73af1-224f-4b01-8b7f-7beeb4565034 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: service: mirror.1 on trial017 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:15:25 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: peer_sites: 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: name: cluster1-DNE 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: state: up+stopped 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: description: local image is primary 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: last_update: 2026-02-22 20:15:28 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: snapshots: 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 196 .mirror.primary.fff73af1-224f-4b01-8b7f-7beeb4565034.40989be1-bd3f-4b5e-848e-8339a6012054 (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459]) 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: 198 .mirror.primary.fff73af1-224f-4b01-8b7f-7beeb4565034.efbde5d1-e215-4032-8314-be2ea5b480ab (peer_uuids:[e056ba3a-4324-4790-8cdf-40b7035d9459])' 2026-02-22T20:15:29.043 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ state: .*up+stopped' 2026-02-22T20:15:29.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: state: up+stopped 2026-02-22T20:15:29.044 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^ description: .*' 2026-02-22T20:15:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: description: local image is primary 2026-02-22T20:15:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n '' ']' 2026-02-22T20:15:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:15:29.046 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep '^up+' 2026-02-22T20:15:29.047 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:15:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep 'service: *mirror..* on ' 2026-02-22T20:15:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: service: mirror.1 on trial017 2026-02-22T20:15:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update 2026-02-22T20:15:29.049 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-22T20:15:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:15:25' 2026-02-22T20:15:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote_user up+stopped '2026-02-22 20:15:25' 2026-02-22T20:15:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:15:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local state_pattern=up+stopped 2026-02-22T20:15:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local 'prev_last_update=2026-02-22 20:15:25' 2026-02-22T20:15:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local status 2026-02-22T20:15:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ CEPH_ARGS= 2026-02-22T20:15:29.051 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2026-02-22T20:15:29.093 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ status='OKOKOK161696176mirror.0trial01719.2.3-689-gf90f4eaetrueOK61736185mirror.1trial01719.2.3-689-gf90f4eaefalseOK61806190mirror.2trial01719.2.3-689-gf90f4eaefalseOK61876193mirror.3trial01719.2.3-689-gf90f4eaefalseOKtest_force_promote_userfff73af1-224f-4b01-8b7f-7beeb4565034up+stoppedlocal image is primary61736185mirror.1trial0172026-02-22 20:15:25cluster1-DNEfbff4f9f-010a-453b-8e51-49f41173e9e5up+stoppedlocal image is primary2026-02-22 20:15:28' 2026-02-22T20:15:29.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local last_update state 2026-02-22T20:15:29.094 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/last_update' 2026-02-22T20:15:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ last_update='2026-02-22 20:15:25' 2026-02-22T20:15:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote_user'\'']/state' 2026-02-22T20:15:29.100 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ state=up+stopped 2026-02-22T20:15:29.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo up+stopped 2026-02-22T20:15:29.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ grep up+stopped 2026-02-22T20:15:29.101 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:up+stopped 2026-02-22T20:15:29.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:15:29.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:15:29.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster1 mirror test_force_promote_user 100 2026-02-22T20:15:29.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:15:29.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:29.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:29.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:15:29.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:15:29.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:15:29.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:15:29.102 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster1 bench mirror/test_force_promote_user --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:15:29.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:15:29.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 6250.15 bytes/sec: 24 MiB/s 2026-02-22T20:15:29.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ write_image cluster2 mirror test_force_promote_user 100 2026-02-22T20:15:29.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:15:29.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:29.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:29.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local count=100 2026-02-22T20:15:29.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local size= 2026-02-22T20:15:29.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:15:29.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ size=4096 2026-02-22T20:15:29.173 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote_user --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2026-02-22T20:15:29.270 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2026-02-22T20:15:29.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout: SEC OPS OPS/SEC BYTES/SEC 2026-02-22T20:15:29.318 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:elapsed: 0 ops: 100 ops/sec: 2083.38 bytes/sec: 8.1 MiB/s 2026-02-22T20:15:29.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster1 mirror test_force_promote_user 2026-02-22T20:15:29.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:15:29.330 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:29.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:29.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:29.331 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:15:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster1 mirror test_force_promote_user 2026-02-22T20:15:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:15:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:29.332 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_force_promote_user 2026-02-22T20:15:29.670 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2026-02-22T20:15:29.679 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster1 rm mirror/test_force_promote_user 2026-02-22T20:15:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:15:30.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:15:30.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image_retry cluster2 mirror test_force_promote_user 2026-02-22T20:15:30.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:15:30.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:30.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:30.974 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:30.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:15:30.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ remove_image cluster2 mirror test_force_promote_user 2026-02-22T20:15:30.975 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:15:30.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pool=mirror 2026-02-22T20:15:30.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local image=test_force_promote_user 2026-02-22T20:15:30.976 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_force_promote_user 2026-02-22T20:15:32.020 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2026-02-22T20:15:32.032 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rbd --cluster=cluster2 rm mirror/test_force_promote_user 2026-02-22T20:15:34.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-22T20:15:34.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ return 0 2026-02-22T20:15:34.668 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ cleanup 0 2026-02-22T20:15:34.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local error_code=0 2026-02-22T20:15:34.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ set +e 2026-02-22T20:15:34.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -ne 0 ']' 2026-02-22T20:15:34.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z '' ']' 2026-02-22T20:15:34.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster instance 2026-02-22T20:15:34.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T20:15:34.669 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2026-02-22T20:15:35.707 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:pool 'mirror' does not exist 2026-02-22T20:15:35.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T20:15:35.718 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2026-02-22T20:15:36.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:pool 'mirror' does not exist 2026-02-22T20:15:36.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T20:15:36.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2026-02-22T20:15:36.711 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:pool 'mirror_parent' does not exist 2026-02-22T20:15:36.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ CEPH_ARGS= 2026-02-22T20:15:36.721 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2026-02-22T20:15:37.069 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:pool 'mirror_parent' does not exist 2026-02-22T20:15:37.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2026-02-22T20:15:37.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirrors cluster1 2026-02-22T20:15:37.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1 2026-02-22T20:15:37.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:15:37.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:15:37.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:37.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:0 '' 2026-02-22T20:15:37.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:0 2026-02-22T20:15:37.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:15:37.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:15:37.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:0 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:0 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:0 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:15:37.082 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:15:37.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:15:37.095 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:15:37.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=59309 2026-02-22T20:15:37.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 59309 ']' 2026-02-22T20:15:37.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 59309 2026-02-22T20:15:37.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:37.097 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:15:37.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:37.098 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=59309 '$2 == pid {print; exit 1}' 2026-02-22T20:15:37.133 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 59309 0.2 0.0 701368 38628 ? Ssl 20:14 0:00 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:15:37.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:37.134 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 1 2026-02-22T20:15:38.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:38.135 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=59309 '$2 == pid {print; exit 1}' 2026-02-22T20:15:38.169 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:ubuntu 59309 0.2 0.0 701368 38628 ? Ssl 20:14 0:00 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2026-02-22T20:15:38.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:38.170 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 2 2026-02-22T20:15:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.171 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=59309 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.206 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:15:40.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.207 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=59309 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster1 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:40.245 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:40.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:40.258 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:0 cluster2 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:0 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-22T20:15:40.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:40.272 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:0 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:0 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:0 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:40.274 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2026-02-22T20:15:40.286 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:15:40.287 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2026-02-22T20:15:40.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:40.288 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:1 '' 2026-02-22T20:15:40.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:1 2026-02-22T20:15:40.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:15:40.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:15:40.289 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:15:40.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:1 2026-02-22T20:15:40.290 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:1 2026-02-22T20:15:40.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:40.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:40.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:1 2026-02-22T20:15:40.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:15:40.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:40.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:40.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:40.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:40.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:15:40.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:15:40.291 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:15:40.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:15:40.292 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:15:40.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:15:40.300 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:15:40.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=59337 2026-02-22T20:15:40.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 59337 ']' 2026-02-22T20:15:40.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 59337 2026-02-22T20:15:40.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:40.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:15:40.302 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.303 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=59337 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:15:40.337 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.338 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=59337 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster1 2026-02-22T20:15:40.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:40.372 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:15:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:15:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:15:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:40.373 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:15:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:40.374 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:40.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:40.384 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:40.386 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:1 cluster2 2026-02-22T20:15:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:1 2026-02-22T20:15:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:15:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-22T20:15:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:15:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:15:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:40.387 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:40.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:40.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:15:40.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:40.388 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-22T20:15:40.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:40.399 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-22T20:15:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:1 2026-02-22T20:15:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:1 2026-02-22T20:15:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-22T20:15:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:1 2026-02-22T20:15:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.401 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:15:40.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-22T20:15:40.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:40.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:40.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:15:40.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:40.402 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2026-02-22T20:15:40.412 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:15:40.413 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2026-02-22T20:15:40.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:40.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:2 '' 2026-02-22T20:15:40.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:2 2026-02-22T20:15:40.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:15:40.415 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:15:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:15:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:2 2026-02-22T20:15:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:2 2026-02-22T20:15:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:2 2026-02-22T20:15:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:15:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:40.416 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:40.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:15:40.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:15:40.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:15:40.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:15:40.417 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:15:40.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:15:40.426 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=59380 2026-02-22T20:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 59380 ']' 2026-02-22T20:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 59380 2026-02-22T20:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:40.428 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:15:40.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.429 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=59380 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:15:40.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.469 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=59380 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.504 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster1 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:40.516 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:40.517 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:2 cluster2 2026-02-22T20:15:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:2 2026-02-22T20:15:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:15:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-22T20:15:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:15:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:15:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:15:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:40.519 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-22T20:15:40.529 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:40.530 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-22T20:15:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:2 2026-02-22T20:15:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:2 2026-02-22T20:15:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-22T20:15:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:2 2026-02-22T20:15:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.532 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:15:40.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-22T20:15:40.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:40.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:40.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:15:40.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:40.533 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2026-02-22T20:15:40.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:15:40.545 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2026-02-22T20:15:40.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:40.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster1:3 '' 2026-02-22T20:15:40.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster1:3 2026-02-22T20:15:40.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:15:40.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:15:40.547 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster1:3 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster1:3 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster1:3 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster1 2026-02-22T20:15:40.548 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster1 2026-02-22T20:15:40.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:15:40.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:15:40.549 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:15:40.558 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:15:40.559 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:15:40.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=59457 2026-02-22T20:15:40.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 59457 ']' 2026-02-22T20:15:40.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 59457 2026-02-22T20:15:40.560 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:15:40.561 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.562 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=59457 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.596 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:15:40.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.597 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=59457 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster1 2026-02-22T20:15:40.632 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:15:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:15:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:15:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:15:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:15:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:40.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:15:40.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:40.634 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:15:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:40.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster1:3 cluster2 2026-02-22T20:15:40.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster1:3 2026-02-22T20:15:40.645 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster1 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster1 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:15:40.646 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:40.647 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-22T20:15:40.657 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:40.658 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-22T20:15:40.660 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster1:3 2026-02-22T20:15:40.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1:3 2026-02-22T20:15:40.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-22T20:15:40.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster1:3 2026-02-22T20:15:40.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:40.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster1 2026-02-22T20:15:40.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:15:40.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-22T20:15:40.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster1 2026-02-22T20:15:40.661 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster1 2026-02-22T20:15:40.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:15:40.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:40.662 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2026-02-22T20:15:40.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:15:40.671 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2026-02-22T20:15:40.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2026-02-22T20:15:40.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirrors cluster2 2026-02-22T20:15:40.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2 2026-02-22T20:15:40.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:15:40.672 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ seq 0 3 2026-02-22T20:15:40.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:40.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster2:0 '' 2026-02-22T20:15:40.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:0 2026-02-22T20:15:40.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:15:40.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:15:40.674 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:15:40.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster2:0 2026-02-22T20:15:40.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2:0 2026-02-22T20:15:40.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:40.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:15:40.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:0 2026-02-22T20:15:40.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:15:40.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:40.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:40.676 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:40.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 0 = cluster2:0 ']' 2026-02-22T20:15:40.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster2 2026-02-22T20:15:40.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster2 2026-02-22T20:15:40.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=0 2026-02-22T20:15:40.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=0 2026-02-22T20:15:40.677 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2026-02-22T20:15:40.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-22T20:15:40.685 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-22T20:15:40.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=55380 2026-02-22T20:15:40.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 55380 ']' 2026-02-22T20:15:40.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 55380 2026-02-22T20:15:40.687 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:40.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:15:40.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.688 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=55380 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:15:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.730 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=55380 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.765 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:0 cluster1 2026-02-22T20:15:40.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:0 2026-02-22T20:15:40.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:15:40.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:15:40.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:15:40.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:40.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:15:40.766 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:15:40.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:15:40.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:15:40.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:15:40.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:40.767 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:15:40.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:15:40.778 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:15:40.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:0 cluster2 2026-02-22T20:15:40.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:0 2026-02-22T20:15:40.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:15:40.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2026-02-22T20:15:40.780 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:15:40.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:40.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:15:40.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:15:40.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:15:40.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:15:40.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:15:40.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:40.781 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2026-02-22T20:15:40.791 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:15:40.792 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2026-02-22T20:15:40.793 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster2:0 2026-02-22T20:15:40.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:0 2026-02-22T20:15:40.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:0 cluster instance 2026-02-22T20:15:40.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:0 2026-02-22T20:15:40.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:40.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:40.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=0 2026-02-22T20:15:40.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 0 = cluster2:0 ']' 2026-02-22T20:15:40.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:15:40.794 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:40.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=0 2026-02-22T20:15:40.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=0 2026-02-22T20:15:40.795 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2026-02-22T20:15:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-22T20:15:40.806 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2026-02-22T20:15:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster2:1 '' 2026-02-22T20:15:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:1 2026-02-22T20:15:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:15:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:15:40.807 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:15:40.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster2:1 2026-02-22T20:15:40.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2:1 2026-02-22T20:15:40.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:40.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:15:40.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:1 2026-02-22T20:15:40.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:15:40.809 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:40.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:40.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:40.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 1 = cluster2:1 ']' 2026-02-22T20:15:40.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster2 2026-02-22T20:15:40.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster2 2026-02-22T20:15:40.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=1 2026-02-22T20:15:40.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=1 2026-02-22T20:15:40.810 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2026-02-22T20:15:40.818 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-22T20:15:40.819 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-22T20:15:40.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=55408 2026-02-22T20:15:40.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 55408 ']' 2026-02-22T20:15:40.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 55408 2026-02-22T20:15:40.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:40.820 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:15:40.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.821 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=55408 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:15:40.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.853 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=55408 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:1 cluster1 2026-02-22T20:15:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:1 2026-02-22T20:15:40.899 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:15:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:15:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:15:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:15:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:15:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:15:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:15:40.900 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:15:40.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:40.901 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:15:40.910 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:15:40.911 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:15:40.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:1 cluster2 2026-02-22T20:15:40.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:1 2026-02-22T20:15:40.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:15:40.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2026-02-22T20:15:40.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:15:40.913 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:40.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:40.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:15:40.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:15:40.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:15:40.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:15:40.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:15:40.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:40.914 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2026-02-22T20:15:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:15:40.924 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2026-02-22T20:15:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster2:1 2026-02-22T20:15:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:1 2026-02-22T20:15:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:1 cluster instance 2026-02-22T20:15:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:1 2026-02-22T20:15:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:40.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:40.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:40.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=1 2026-02-22T20:15:40.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 1 = cluster2:1 ']' 2026-02-22T20:15:40.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:15:40.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:40.927 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=1 2026-02-22T20:15:40.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=1 2026-02-22T20:15:40.928 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2026-02-22T20:15:40.938 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-22T20:15:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2026-02-22T20:15:40.939 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster2:2 '' 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:2 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster2:2 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2:2 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:2 cluster instance 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:2 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 2 = cluster2:2 ']' 2026-02-22T20:15:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster2 2026-02-22T20:15:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster2 2026-02-22T20:15:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=2 2026-02-22T20:15:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=2 2026-02-22T20:15:40.941 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2026-02-22T20:15:40.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-22T20:15:40.951 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-22T20:15:40.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=55452 2026-02-22T20:15:40.952 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 55452 ']' 2026-02-22T20:15:40.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 55452 2026-02-22T20:15:40.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:40.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:15:40.953 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.954 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=55452 '$2 == pid {print; exit 1}' 2026-02-22T20:15:40.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:15:40.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:40.989 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=55452 '$2 == pid {print; exit 1}' 2026-02-22T20:15:41.023 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:2 cluster1 2026-02-22T20:15:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:2 2026-02-22T20:15:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:15:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-22T20:15:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:15:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:41.024 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:15:41.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:15:41.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:15:41.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:15:41.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:15:41.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:41.025 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-22T20:15:41.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:15:41.035 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:15:41.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:2 cluster2 2026-02-22T20:15:41.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:2 2026-02-22T20:15:41.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:15:41.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:41.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 local_cluster instance 2026-02-22T20:15:41.037 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:15:41.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:41.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:41.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:41.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:15:41.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:15:41.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:15:41.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:15:41.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:15:41.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:41.038 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket' 2026-02-22T20:15:41.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:15:41.052 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2026-02-22T20:15:41.054 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster2:2 2026-02-22T20:15:41.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:2 2026-02-22T20:15:41.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:41.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:2 cluster instance 2026-02-22T20:15:41.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:2 2026-02-22T20:15:41.055 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=2 2026-02-22T20:15:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 2 = cluster2:2 ']' 2026-02-22T20:15:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:15:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:41.056 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=2 2026-02-22T20:15:41.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=2 2026-02-22T20:15:41.057 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2026-02-22T20:15:41.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-22T20:15:41.067 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2026-02-22T20:15:41.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-22T20:15:41.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ stop_mirror cluster2:3 '' 2026-02-22T20:15:41.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local cluster=cluster2:3 2026-02-22T20:15:41.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local sig= 2026-02-22T20:15:41.070 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test -n '' 2026-02-22T20:15:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ local pid 2026-02-22T20:15:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ daemon_pid_file cluster2:3 2026-02-22T20:15:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster=cluster2:3 2026-02-22T20:15:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance 2026-02-22T20:15:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ set_cluster_instance cluster2:3 cluster instance 2026-02-22T20:15:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local val=cluster2:3 2026-02-22T20:15:41.071 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local cluster_var_name=cluster 2026-02-22T20:15:41.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local instance_var_name=instance 2026-02-22T20:15:41.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:41.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:41.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ '[' 3 = cluster2:3 ']' 2026-02-22T20:15:41.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval cluster=cluster2 2026-02-22T20:15:41.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ cluster=cluster2 2026-02-22T20:15:41.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ eval instance=3 2026-02-22T20:15:41.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ instance=3 2026-02-22T20:15:41.072 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2026-02-22T20:15:41.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-22T20:15:41.079 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-22T20:15:41.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ pid=55512 2026-02-22T20:15:41.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -n 55512 ']' 2026-02-22T20:15:41.080 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ kill 55512 2026-02-22T20:15:41.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-22T20:15:41.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ sleep 0 2026-02-22T20:15:41.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:41.081 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=55512 '$2 == pid {print; exit 1}' 2026-02-22T20:15:41.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ break 2026-02-22T20:15:41.116 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ ps auxww 2026-02-22T20:15:41.117 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ awk -v pid=55512 '$2 == pid {print; exit 1}' 2026-02-22T20:15:41.150 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:3 cluster1 2026-02-22T20:15:41.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:3 2026-02-22T20:15:41.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster1 2026-02-22T20:15:41.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:41.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-22T20:15:41.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:15:41.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:41.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:41.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:41.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:15:41.151 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:15:41.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:15:41.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:15:41.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:15:41.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:41.152 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-22T20:15:41.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:15:41.163 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:15:41.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_asok_file cluster2:3 cluster2 2026-02-22T20:15:41.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local local_cluster=cluster2:3 2026-02-22T20:15:41.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2 2026-02-22T20:15:41.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:41.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 local_cluster instance 2026-02-22T20:15:41.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:15:41.165 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=local_cluster 2026-02-22T20:15:41.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:41.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:41.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:15:41.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:15:41.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval local_cluster=cluster2 2026-02-22T20:15:41.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ local_cluster=cluster2 2026-02-22T20:15:41.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:15:41.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:41.166 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket' 2026-02-22T20:15:41.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:15:41.176 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2026-02-22T20:15:41.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ daemon_pid_file cluster2:3 2026-02-22T20:15:41.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster=cluster2:3 2026-02-22T20:15:41.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance 2026-02-22T20:15:41.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ set_cluster_instance cluster2:3 cluster instance 2026-02-22T20:15:41.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local val=cluster2:3 2026-02-22T20:15:41.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local cluster_var_name=cluster 2026-02-22T20:15:41.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ local instance_var_name=instance 2026-02-22T20:15:41.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ cluster=cluster2 2026-02-22T20:15:41.179 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ instance=3 2026-02-22T20:15:41.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ '[' 3 = cluster2:3 ']' 2026-02-22T20:15:41.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval cluster=cluster2 2026-02-22T20:15:41.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ cluster=cluster2 2026-02-22T20:15:41.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ eval instance=3 2026-02-22T20:15:41.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ instance=3 2026-02-22T20:15:41.180 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2026-02-22T20:15:41.190 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-22T20:15:41.191 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2026-02-22T20:15:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' -z 1 ']' 2026-02-22T20:15:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ test '' = /tmp/tmp.bhaDYZBKee 2026-02-22T20:15:41.192 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ rm -Rf /tmp/tmp.bhaDYZBKee 2026-02-22T20:15:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ '[' 0 -eq 0 ']' 2026-02-22T20:15:41.194 INFO:tasks.workunit.cluster1.client.mirror.trial017.stdout:OK 2026-02-22T20:15:41.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ echo OK 2026-02-22T20:15:41.195 INFO:tasks.workunit.cluster1.client.mirror.trial017.stderr:+ exit 0 2026-02-22T20:15:41.195 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T20:15:41.195 DEBUG:teuthology.orchestra.run.trial017:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2026-02-22T20:15:41.244 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror.sh'] on cluster1.client.mirror... 2026-02-22T20:15:41.244 DEBUG:teuthology.orchestra.run.trial017:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2026-02-22T20:15:41.580 DEBUG:teuthology.parallel:result is None 2026-02-22T20:15:41.580 DEBUG:teuthology.orchestra.run.trial017:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-22T20:15:41.590 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-22T20:15:41.590 DEBUG:teuthology.orchestra.run.trial017:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-22T20:15:41.640 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-22T20:15:41.640 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-22T20:15:41.650 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2026-02-22T20:15:41.650 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T20:15:41.831 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T20:15:41.831 INFO:teuthology.orchestra.run.trial017.stderr:dumped all 2026-02-22T20:15:41.841 INFO:teuthology.orchestra.run.trial017.stdout:{"pg_ready":true,"pg_map":{"version":1195,"stamp":"2026-02-22T20:15:40.943548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":861784,"kb_used_data":12000,"kb_used_omap":4,"kb_used_meta":849723,"kb_avail":282253736,"statfs":{"total":289910292480,"available":289027825664,"internally_reserved":0,"allocated":12288000,"data_stored":7729959,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":870116691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7262752423,"num_objects":-1934,"num_object_clones":-141,"num_object_copies":-3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1934,"num_whiteouts":0,"num_read":-131115,"num_read_kb":-15382415,"num_write":-43697,"num_write_kb":-46236811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.932679"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":982,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026375+0000","last_change":"2026-02-22T19:52:24.779439+0000","last_active":"2026-02-22T20:15:34.026375+0000","last_peered":"2026-02-22T20:15:34.026375+0000","last_clean":"2026-02-22T20:15:34.026375+0000","last_became_active":"2026-02-22T19:52:22.776103+0000","last_became_peered":"2026-02-22T19:52:22.776103+0000","last_unstale":"2026-02-22T20:15:34.026375+0000","last_undegraded":"2026-02-22T20:15:34.026375+0000","last_fullsized":"2026-02-22T20:15:34.026375+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:02:03.695406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044655200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":982,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026875+0000","last_change":"2026-02-22T19:52:24.779510+0000","last_active":"2026-02-22T20:15:34.026875+0000","last_peered":"2026-02-22T20:15:34.026875+0000","last_clean":"2026-02-22T20:15:34.026875+0000","last_became_active":"2026-02-22T19:52:22.776089+0000","last_became_peered":"2026-02-22T19:52:22.776089+0000","last_unstale":"2026-02-22T20:15:34.026875+0000","last_undegraded":"2026-02-22T20:15:34.026875+0000","last_fullsized":"2026-02-22T20:15:34.026875+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:33:54.099391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051971799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":983,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026991+0000","last_change":"2026-02-22T19:52:24.779619+0000","last_active":"2026-02-22T20:15:34.026991+0000","last_peered":"2026-02-22T20:15:34.026991+0000","last_clean":"2026-02-22T20:15:34.026991+0000","last_became_active":"2026-02-22T19:52:22.775967+0000","last_became_peered":"2026-02-22T19:52:22.775967+0000","last_unstale":"2026-02-22T20:15:34.026991+0000","last_undegraded":"2026-02-22T20:15:34.026991+0000","last_fullsized":"2026-02-22T20:15:34.026991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:57:59.097058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060316299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":983,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.025449+0000","last_change":"2026-02-22T19:52:24.779289+0000","last_active":"2026-02-22T20:15:34.025449+0000","last_peered":"2026-02-22T20:15:34.025449+0000","last_clean":"2026-02-22T20:15:34.025449+0000","last_became_active":"2026-02-22T19:52:22.775149+0000","last_became_peered":"2026-02-22T19:52:22.775149+0000","last_unstale":"2026-02-22T20:15:34.025449+0000","last_undegraded":"2026-02-22T20:15:34.025449+0000","last_fullsized":"2026-02-22T20:15:34.025449+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:14:32.713489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033844000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":986,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.022903+0000","last_change":"2026-02-22T19:52:24.777066+0000","last_active":"2026-02-22T20:15:34.022903+0000","last_peered":"2026-02-22T20:15:34.022903+0000","last_clean":"2026-02-22T20:15:34.022903+0000","last_became_active":"2026-02-22T19:52:22.775845+0000","last_became_peered":"2026-02-22T19:52:22.775845+0000","last_unstale":"2026-02-22T20:15:34.022903+0000","last_undegraded":"2026-02-22T20:15:34.022903+0000","last_fullsized":"2026-02-22T20:15:34.022903+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:44:39.886032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033930199999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":986,"reported_epoch":494,"state":"active+clean","last_fresh":"2026-02-22T20:15:36.032924+0000","last_change":"2026-02-22T19:52:25.383915+0000","last_active":"2026-02-22T20:15:36.032924+0000","last_peered":"2026-02-22T20:15:36.032924+0000","last_clean":"2026-02-22T20:15:36.032924+0000","last_became_active":"2026-02-22T19:52:22.775858+0000","last_became_peered":"2026-02-22T19:52:22.775858+0000","last_unstale":"2026-02-22T20:15:36.032924+0000","last_undegraded":"2026-02-22T20:15:36.032924+0000","last_fullsized":"2026-02-22T20:15:36.032924+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:35:20.222151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035074299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":986,"reported_epoch":494,"state":"active+clean","last_fresh":"2026-02-22T20:15:36.033672+0000","last_change":"2026-02-22T19:52:25.383852+0000","last_active":"2026-02-22T20:15:36.033672+0000","last_peered":"2026-02-22T20:15:36.033672+0000","last_clean":"2026-02-22T20:15:36.033672+0000","last_became_active":"2026-02-22T19:52:22.775507+0000","last_became_peered":"2026-02-22T19:52:22.775507+0000","last_unstale":"2026-02-22T20:15:36.033672+0000","last_undegraded":"2026-02-22T20:15:36.033672+0000","last_fullsized":"2026-02-22T20:15:36.033672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:52:30.344481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030984600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":984,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026064+0000","last_change":"2026-02-22T19:52:24.779287+0000","last_active":"2026-02-22T20:15:34.026064+0000","last_peered":"2026-02-22T20:15:34.026064+0000","last_clean":"2026-02-22T20:15:34.026064+0000","last_became_active":"2026-02-22T19:52:22.774567+0000","last_became_peered":"2026-02-22T19:52:22.774567+0000","last_unstale":"2026-02-22T20:15:34.026064+0000","last_undegraded":"2026-02-22T20:15:34.026064+0000","last_fullsized":"2026-02-22T20:15:34.026064+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:41:09.322918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036364699999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'72","reported_seq":1074,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.025313+0000","last_change":"2026-02-22T19:52:18.864376+0000","last_active":"2026-02-22T20:15:34.025313+0000","last_peered":"2026-02-22T20:15:34.025313+0000","last_clean":"2026-02-22T20:15:34.025313+0000","last_became_active":"2026-02-22T19:52:18.863570+0000","last_became_peered":"2026-02-22T19:52:18.863570+0000","last_unstale":"2026-02-22T20:15:34.025313+0000","last_undegraded":"2026-02-22T20:15:34.025313+0000","last_fullsized":"2026-02-22T20:15:34.025313+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_clean_scrub_stamp":"2026-02-22T19:52:17.758264+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:09:52.495537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738652,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269700,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":265662,"kb_avail":94102140,"statfs":{"total":96636764160,"available":96360591360,"internally_reserved":0,"allocated":4112384,"data_stored":2772072,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":272038342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738650,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314112,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":310078,"kb_avail":94057728,"statfs":{"total":96636764160,"available":96315113472,"internally_reserved":0,"allocated":4112384,"data_stored":2772072,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":317520327},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738651,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":277972,"kb_used_data":3968,"kb_used_omap":1,"kb_used_meta":273982,"kb_avail":94093868,"statfs":{"total":96636764160,"available":96352120832,"internally_reserved":0,"allocated":4063232,"data_stored":2185815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280558022},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:15:41.842 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T20:15:41.991 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T20:15:41.992 INFO:teuthology.orchestra.run.trial017.stderr:dumped all 2026-02-22T20:15:42.001 INFO:teuthology.orchestra.run.trial017.stdout:{"pg_ready":true,"pg_map":{"version":1195,"stamp":"2026-02-22T20:15:40.943548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":861784,"kb_used_data":12000,"kb_used_omap":4,"kb_used_meta":849723,"kb_avail":282253736,"statfs":{"total":289910292480,"available":289027825664,"internally_reserved":0,"allocated":12288000,"data_stored":7729959,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":870116691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7262752423,"num_objects":-1934,"num_object_clones":-141,"num_object_copies":-3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1934,"num_whiteouts":0,"num_read":-131115,"num_read_kb":-15382415,"num_write":-43697,"num_write_kb":-46236811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.932679"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":982,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026375+0000","last_change":"2026-02-22T19:52:24.779439+0000","last_active":"2026-02-22T20:15:34.026375+0000","last_peered":"2026-02-22T20:15:34.026375+0000","last_clean":"2026-02-22T20:15:34.026375+0000","last_became_active":"2026-02-22T19:52:22.776103+0000","last_became_peered":"2026-02-22T19:52:22.776103+0000","last_unstale":"2026-02-22T20:15:34.026375+0000","last_undegraded":"2026-02-22T20:15:34.026375+0000","last_fullsized":"2026-02-22T20:15:34.026375+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:02:03.695406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044655200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":982,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026875+0000","last_change":"2026-02-22T19:52:24.779510+0000","last_active":"2026-02-22T20:15:34.026875+0000","last_peered":"2026-02-22T20:15:34.026875+0000","last_clean":"2026-02-22T20:15:34.026875+0000","last_became_active":"2026-02-22T19:52:22.776089+0000","last_became_peered":"2026-02-22T19:52:22.776089+0000","last_unstale":"2026-02-22T20:15:34.026875+0000","last_undegraded":"2026-02-22T20:15:34.026875+0000","last_fullsized":"2026-02-22T20:15:34.026875+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:33:54.099391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051971799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":983,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026991+0000","last_change":"2026-02-22T19:52:24.779619+0000","last_active":"2026-02-22T20:15:34.026991+0000","last_peered":"2026-02-22T20:15:34.026991+0000","last_clean":"2026-02-22T20:15:34.026991+0000","last_became_active":"2026-02-22T19:52:22.775967+0000","last_became_peered":"2026-02-22T19:52:22.775967+0000","last_unstale":"2026-02-22T20:15:34.026991+0000","last_undegraded":"2026-02-22T20:15:34.026991+0000","last_fullsized":"2026-02-22T20:15:34.026991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:57:59.097058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060316299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":983,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.025449+0000","last_change":"2026-02-22T19:52:24.779289+0000","last_active":"2026-02-22T20:15:34.025449+0000","last_peered":"2026-02-22T20:15:34.025449+0000","last_clean":"2026-02-22T20:15:34.025449+0000","last_became_active":"2026-02-22T19:52:22.775149+0000","last_became_peered":"2026-02-22T19:52:22.775149+0000","last_unstale":"2026-02-22T20:15:34.025449+0000","last_undegraded":"2026-02-22T20:15:34.025449+0000","last_fullsized":"2026-02-22T20:15:34.025449+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:14:32.713489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033844000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":986,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.022903+0000","last_change":"2026-02-22T19:52:24.777066+0000","last_active":"2026-02-22T20:15:34.022903+0000","last_peered":"2026-02-22T20:15:34.022903+0000","last_clean":"2026-02-22T20:15:34.022903+0000","last_became_active":"2026-02-22T19:52:22.775845+0000","last_became_peered":"2026-02-22T19:52:22.775845+0000","last_unstale":"2026-02-22T20:15:34.022903+0000","last_undegraded":"2026-02-22T20:15:34.022903+0000","last_fullsized":"2026-02-22T20:15:34.022903+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:44:39.886032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033930199999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":986,"reported_epoch":494,"state":"active+clean","last_fresh":"2026-02-22T20:15:36.032924+0000","last_change":"2026-02-22T19:52:25.383915+0000","last_active":"2026-02-22T20:15:36.032924+0000","last_peered":"2026-02-22T20:15:36.032924+0000","last_clean":"2026-02-22T20:15:36.032924+0000","last_became_active":"2026-02-22T19:52:22.775858+0000","last_became_peered":"2026-02-22T19:52:22.775858+0000","last_unstale":"2026-02-22T20:15:36.032924+0000","last_undegraded":"2026-02-22T20:15:36.032924+0000","last_fullsized":"2026-02-22T20:15:36.032924+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:35:20.222151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035074299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":986,"reported_epoch":494,"state":"active+clean","last_fresh":"2026-02-22T20:15:36.033672+0000","last_change":"2026-02-22T19:52:25.383852+0000","last_active":"2026-02-22T20:15:36.033672+0000","last_peered":"2026-02-22T20:15:36.033672+0000","last_clean":"2026-02-22T20:15:36.033672+0000","last_became_active":"2026-02-22T19:52:22.775507+0000","last_became_peered":"2026-02-22T19:52:22.775507+0000","last_unstale":"2026-02-22T20:15:36.033672+0000","last_undegraded":"2026-02-22T20:15:36.033672+0000","last_fullsized":"2026-02-22T20:15:36.033672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:52:30.344481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030984600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":984,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026064+0000","last_change":"2026-02-22T19:52:24.779287+0000","last_active":"2026-02-22T20:15:34.026064+0000","last_peered":"2026-02-22T20:15:34.026064+0000","last_clean":"2026-02-22T20:15:34.026064+0000","last_became_active":"2026-02-22T19:52:22.774567+0000","last_became_peered":"2026-02-22T19:52:22.774567+0000","last_unstale":"2026-02-22T20:15:34.026064+0000","last_undegraded":"2026-02-22T20:15:34.026064+0000","last_fullsized":"2026-02-22T20:15:34.026064+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:41:09.322918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036364699999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'72","reported_seq":1074,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.025313+0000","last_change":"2026-02-22T19:52:18.864376+0000","last_active":"2026-02-22T20:15:34.025313+0000","last_peered":"2026-02-22T20:15:34.025313+0000","last_clean":"2026-02-22T20:15:34.025313+0000","last_became_active":"2026-02-22T19:52:18.863570+0000","last_became_peered":"2026-02-22T19:52:18.863570+0000","last_unstale":"2026-02-22T20:15:34.025313+0000","last_undegraded":"2026-02-22T20:15:34.025313+0000","last_fullsized":"2026-02-22T20:15:34.025313+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_clean_scrub_stamp":"2026-02-22T19:52:17.758264+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:09:52.495537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738652,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269700,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":265662,"kb_avail":94102140,"statfs":{"total":96636764160,"available":96360591360,"internally_reserved":0,"allocated":4112384,"data_stored":2772072,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":272038342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738650,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314112,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":310078,"kb_avail":94057728,"statfs":{"total":96636764160,"available":96315113472,"internally_reserved":0,"allocated":4112384,"data_stored":2772072,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":317520327},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738651,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":277972,"kb_used_data":3968,"kb_used_omap":1,"kb_used_meta":273982,"kb_avail":94093868,"statfs":{"total":96636764160,"available":96352120832,"internally_reserved":0,"allocated":4063232,"data_stored":2185815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280558022},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:15:42.003 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2026-02-22T20:15:42.003 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T20:15:42.155 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T20:15:42.156 INFO:teuthology.orchestra.run.trial017.stderr:dumped all 2026-02-22T20:15:42.166 INFO:teuthology.orchestra.run.trial017.stdout:{"pg_ready":true,"pg_map":{"version":1195,"stamp":"2026-02-22T20:15:40.943548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":861784,"kb_used_data":12000,"kb_used_omap":4,"kb_used_meta":849723,"kb_avail":282253736,"statfs":{"total":289910292480,"available":289027825664,"internally_reserved":0,"allocated":12288000,"data_stored":7729959,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":870116691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7262752423,"num_objects":-1934,"num_object_clones":-141,"num_object_copies":-3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1934,"num_whiteouts":0,"num_read":-131115,"num_read_kb":-15382415,"num_write":-43697,"num_write_kb":-46236811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.932679"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":982,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026375+0000","last_change":"2026-02-22T19:52:24.779439+0000","last_active":"2026-02-22T20:15:34.026375+0000","last_peered":"2026-02-22T20:15:34.026375+0000","last_clean":"2026-02-22T20:15:34.026375+0000","last_became_active":"2026-02-22T19:52:22.776103+0000","last_became_peered":"2026-02-22T19:52:22.776103+0000","last_unstale":"2026-02-22T20:15:34.026375+0000","last_undegraded":"2026-02-22T20:15:34.026375+0000","last_fullsized":"2026-02-22T20:15:34.026375+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:02:03.695406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044655200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":982,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026875+0000","last_change":"2026-02-22T19:52:24.779510+0000","last_active":"2026-02-22T20:15:34.026875+0000","last_peered":"2026-02-22T20:15:34.026875+0000","last_clean":"2026-02-22T20:15:34.026875+0000","last_became_active":"2026-02-22T19:52:22.776089+0000","last_became_peered":"2026-02-22T19:52:22.776089+0000","last_unstale":"2026-02-22T20:15:34.026875+0000","last_undegraded":"2026-02-22T20:15:34.026875+0000","last_fullsized":"2026-02-22T20:15:34.026875+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:33:54.099391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051971799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":983,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026991+0000","last_change":"2026-02-22T19:52:24.779619+0000","last_active":"2026-02-22T20:15:34.026991+0000","last_peered":"2026-02-22T20:15:34.026991+0000","last_clean":"2026-02-22T20:15:34.026991+0000","last_became_active":"2026-02-22T19:52:22.775967+0000","last_became_peered":"2026-02-22T19:52:22.775967+0000","last_unstale":"2026-02-22T20:15:34.026991+0000","last_undegraded":"2026-02-22T20:15:34.026991+0000","last_fullsized":"2026-02-22T20:15:34.026991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:57:59.097058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060316299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":983,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.025449+0000","last_change":"2026-02-22T19:52:24.779289+0000","last_active":"2026-02-22T20:15:34.025449+0000","last_peered":"2026-02-22T20:15:34.025449+0000","last_clean":"2026-02-22T20:15:34.025449+0000","last_became_active":"2026-02-22T19:52:22.775149+0000","last_became_peered":"2026-02-22T19:52:22.775149+0000","last_unstale":"2026-02-22T20:15:34.025449+0000","last_undegraded":"2026-02-22T20:15:34.025449+0000","last_fullsized":"2026-02-22T20:15:34.025449+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:14:32.713489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033844000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":986,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.022903+0000","last_change":"2026-02-22T19:52:24.777066+0000","last_active":"2026-02-22T20:15:34.022903+0000","last_peered":"2026-02-22T20:15:34.022903+0000","last_clean":"2026-02-22T20:15:34.022903+0000","last_became_active":"2026-02-22T19:52:22.775845+0000","last_became_peered":"2026-02-22T19:52:22.775845+0000","last_unstale":"2026-02-22T20:15:34.022903+0000","last_undegraded":"2026-02-22T20:15:34.022903+0000","last_fullsized":"2026-02-22T20:15:34.022903+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:44:39.886032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033930199999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":986,"reported_epoch":494,"state":"active+clean","last_fresh":"2026-02-22T20:15:36.032924+0000","last_change":"2026-02-22T19:52:25.383915+0000","last_active":"2026-02-22T20:15:36.032924+0000","last_peered":"2026-02-22T20:15:36.032924+0000","last_clean":"2026-02-22T20:15:36.032924+0000","last_became_active":"2026-02-22T19:52:22.775858+0000","last_became_peered":"2026-02-22T19:52:22.775858+0000","last_unstale":"2026-02-22T20:15:36.032924+0000","last_undegraded":"2026-02-22T20:15:36.032924+0000","last_fullsized":"2026-02-22T20:15:36.032924+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:35:20.222151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035074299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":986,"reported_epoch":494,"state":"active+clean","last_fresh":"2026-02-22T20:15:36.033672+0000","last_change":"2026-02-22T19:52:25.383852+0000","last_active":"2026-02-22T20:15:36.033672+0000","last_peered":"2026-02-22T20:15:36.033672+0000","last_clean":"2026-02-22T20:15:36.033672+0000","last_became_active":"2026-02-22T19:52:22.775507+0000","last_became_peered":"2026-02-22T19:52:22.775507+0000","last_unstale":"2026-02-22T20:15:36.033672+0000","last_undegraded":"2026-02-22T20:15:36.033672+0000","last_fullsized":"2026-02-22T20:15:36.033672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:52:30.344481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030984600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":984,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.026064+0000","last_change":"2026-02-22T19:52:24.779287+0000","last_active":"2026-02-22T20:15:34.026064+0000","last_peered":"2026-02-22T20:15:34.026064+0000","last_clean":"2026-02-22T20:15:34.026064+0000","last_became_active":"2026-02-22T19:52:22.774567+0000","last_became_peered":"2026-02-22T19:52:22.774567+0000","last_unstale":"2026-02-22T20:15:34.026064+0000","last_undegraded":"2026-02-22T20:15:34.026064+0000","last_fullsized":"2026-02-22T20:15:34.026064+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:41:09.322918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036364699999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'72","reported_seq":1074,"reported_epoch":493,"state":"active+clean","last_fresh":"2026-02-22T20:15:34.025313+0000","last_change":"2026-02-22T19:52:18.864376+0000","last_active":"2026-02-22T20:15:34.025313+0000","last_peered":"2026-02-22T20:15:34.025313+0000","last_clean":"2026-02-22T20:15:34.025313+0000","last_became_active":"2026-02-22T19:52:18.863570+0000","last_became_peered":"2026-02-22T19:52:18.863570+0000","last_unstale":"2026-02-22T20:15:34.025313+0000","last_undegraded":"2026-02-22T20:15:34.025313+0000","last_fullsized":"2026-02-22T20:15:34.025313+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_clean_scrub_stamp":"2026-02-22T19:52:17.758264+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:09:52.495537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738652,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269700,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":265662,"kb_avail":94102140,"statfs":{"total":96636764160,"available":96360591360,"internally_reserved":0,"allocated":4112384,"data_stored":2772072,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":272038342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738650,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314112,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":310078,"kb_avail":94057728,"statfs":{"total":96636764160,"available":96315113472,"internally_reserved":0,"allocated":4112384,"data_stored":2772072,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":317520327},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738651,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":277972,"kb_used_data":3968,"kb_used_omap":1,"kb_used_meta":273982,"kb_avail":94093868,"statfs":{"total":96636764160,"available":96352120832,"internally_reserved":0,"allocated":4063232,"data_stored":2185815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280558022},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:15:42.167 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-22T20:15:42.322 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T20:15:42.322 INFO:teuthology.orchestra.run.trial017.stdout:{"epoch":495,"fsid":"10943a12-2d65-41e2-884e-f5f3b2ca910b","created":"2026-02-22T19:52:07.064998+0000","modified":"2026-02-22T20:15:37.026709+0000","last_up_change":"2026-02-22T19:52:15.751975+0000","last_in_change":"2026-02-22T19:52:07.621477+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:52:16.778879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:52:21.214161+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bb49012e-bab3-4325-b6fa-8bb3ea30380c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6800","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6801","nonce":2343458682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6802","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6803","nonce":2343458682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6806","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6807","nonce":2343458682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6804","nonce":2343458682},{"type":"v1","addr":"10.20.193.17:6805","nonce":2343458682}]},"public_addr":"10.20.193.17:6801/2343458682","cluster_addr":"10.20.193.17:6803/2343458682","heartbeat_back_addr":"10.20.193.17:6807/2343458682","heartbeat_front_addr":"10.20.193.17:6805/2343458682","state":["exists","up"]},{"osd":1,"uuid":"090c5ffe-2b5d-4ef9-96b9-a5a6e6232d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6816","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6817","nonce":92566604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6818","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6819","nonce":92566604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6822","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6823","nonce":92566604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6820","nonce":92566604},{"type":"v1","addr":"10.20.193.17:6821","nonce":92566604}]},"public_addr":"10.20.193.17:6817/92566604","cluster_addr":"10.20.193.17:6819/92566604","heartbeat_back_addr":"10.20.193.17:6823/92566604","heartbeat_front_addr":"10.20.193.17:6821/92566604","state":["exists","up"]},{"osd":2,"uuid":"e43281b8-07e8-4d2b-90da-609ff2835583","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6808","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6809","nonce":2835130688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6810","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6811","nonce":2835130688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6814","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6815","nonce":2835130688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.17:6812","nonce":2835130688},{"type":"v1","addr":"10.20.193.17:6813","nonce":2835130688}]},"public_addr":"10.20.193.17:6809/2835130688","cluster_addr":"10.20.193.17:6811/2835130688","heartbeat_back_addr":"10.20.193.17:6815/2835130688","heartbeat_front_addr":"10.20.193.17:6813/2835130688","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:13.926503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:14.155040+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:52:14.538654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":196,"length":3}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T20:15:43.334 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2026-02-22T20:15:43.334 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:15:43.417 INFO:teuthology.orchestra.run.trial017.stdout:{ 2026-02-22T20:15:43.418 INFO:teuthology.orchestra.run.trial017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:15:43.418 INFO:teuthology.orchestra.run.trial017.stdout:} 2026-02-22T20:15:43.427 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2026-02-22T20:15:43.578 INFO:teuthology.orchestra.run.trial017.stderr:instructed osd(s) 0 to deep-scrub 2026-02-22T20:15:43.590 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2026-02-22T20:15:43.590 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:15:43.671 INFO:teuthology.orchestra.run.trial017.stdout:{ 2026-02-22T20:15:43.671 INFO:teuthology.orchestra.run.trial017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:15:43.671 INFO:teuthology.orchestra.run.trial017.stdout:} 2026-02-22T20:15:43.680 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2026-02-22T20:15:43.824 INFO:teuthology.orchestra.run.trial017.stderr:instructed osd(s) 1 to deep-scrub 2026-02-22T20:15:43.835 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2026-02-22T20:15:43.835 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:15:43.915 INFO:teuthology.orchestra.run.trial017.stdout:{ 2026-02-22T20:15:43.915 INFO:teuthology.orchestra.run.trial017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T20:15:43.915 INFO:teuthology.orchestra.run.trial017.stdout:} 2026-02-22T20:15:43.924 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2026-02-22T20:15:44.076 INFO:teuthology.orchestra.run.trial017.stderr:instructed osd(s) 2 to deep-scrub 2026-02-22T20:15:44.088 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T20:15:44.227 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T20:15:44.227 INFO:teuthology.orchestra.run.trial017.stderr:dumped all 2026-02-22T20:15:44.238 INFO:teuthology.orchestra.run.trial017.stdout:{"pg_ready":true,"pg_map":{"version":1196,"stamp":"2026-02-22T20:15:42.943822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":196,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":861840,"kb_used_data":12048,"kb_used_omap":4,"kb_used_meta":849723,"kb_avail":282253680,"statfs":{"total":289910292480,"available":289027768320,"internally_reserved":0,"allocated":12337152,"data_stored":7749130,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":870116691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7262752423,"num_objects":-1934,"num_object_clones":-141,"num_object_copies":-3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1934,"num_whiteouts":0,"num_read":-131115,"num_read_kb":-15382415,"num_write":-43697,"num_write_kb":-46236811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000983"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":985,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.983952+0000","last_change":"2026-02-22T19:52:24.779439+0000","last_active":"2026-02-22T20:15:37.983952+0000","last_peered":"2026-02-22T20:15:37.983952+0000","last_clean":"2026-02-22T20:15:37.983952+0000","last_became_active":"2026-02-22T19:52:22.776103+0000","last_became_peered":"2026-02-22T19:52:22.776103+0000","last_unstale":"2026-02-22T20:15:37.983952+0000","last_undegraded":"2026-02-22T20:15:37.983952+0000","last_fullsized":"2026-02-22T20:15:37.983952+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:02:03.695406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044655200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":985,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.988851+0000","last_change":"2026-02-22T19:52:24.779510+0000","last_active":"2026-02-22T20:15:37.988851+0000","last_peered":"2026-02-22T20:15:37.988851+0000","last_clean":"2026-02-22T20:15:37.988851+0000","last_became_active":"2026-02-22T19:52:22.776089+0000","last_became_peered":"2026-02-22T19:52:22.776089+0000","last_unstale":"2026-02-22T20:15:37.988851+0000","last_undegraded":"2026-02-22T20:15:37.988851+0000","last_fullsized":"2026-02-22T20:15:37.988851+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:33:54.099391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051971799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":986,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.989364+0000","last_change":"2026-02-22T19:52:24.779619+0000","last_active":"2026-02-22T20:15:37.989364+0000","last_peered":"2026-02-22T20:15:37.989364+0000","last_clean":"2026-02-22T20:15:37.989364+0000","last_became_active":"2026-02-22T19:52:22.775967+0000","last_became_peered":"2026-02-22T19:52:22.775967+0000","last_unstale":"2026-02-22T20:15:37.989364+0000","last_undegraded":"2026-02-22T20:15:37.989364+0000","last_fullsized":"2026-02-22T20:15:37.989364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:57:59.097058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060316299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":986,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.982060+0000","last_change":"2026-02-22T19:52:24.779289+0000","last_active":"2026-02-22T20:15:37.982060+0000","last_peered":"2026-02-22T20:15:37.982060+0000","last_clean":"2026-02-22T20:15:37.982060+0000","last_became_active":"2026-02-22T19:52:22.775149+0000","last_became_peered":"2026-02-22T19:52:22.775149+0000","last_unstale":"2026-02-22T20:15:37.982060+0000","last_undegraded":"2026-02-22T20:15:37.982060+0000","last_fullsized":"2026-02-22T20:15:37.982060+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:14:32.713489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033844000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":989,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:38.647089+0000","last_change":"2026-02-22T19:52:24.777066+0000","last_active":"2026-02-22T20:15:38.647089+0000","last_peered":"2026-02-22T20:15:38.647089+0000","last_clean":"2026-02-22T20:15:38.647089+0000","last_became_active":"2026-02-22T19:52:22.775845+0000","last_became_peered":"2026-02-22T19:52:22.775845+0000","last_unstale":"2026-02-22T20:15:38.647089+0000","last_undegraded":"2026-02-22T20:15:38.647089+0000","last_fullsized":"2026-02-22T20:15:38.647089+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:44:39.886032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033930199999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":988,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.034410+0000","last_change":"2026-02-22T19:52:25.383915+0000","last_active":"2026-02-22T20:15:37.034410+0000","last_peered":"2026-02-22T20:15:37.034410+0000","last_clean":"2026-02-22T20:15:37.034410+0000","last_became_active":"2026-02-22T19:52:22.775858+0000","last_became_peered":"2026-02-22T19:52:22.775858+0000","last_unstale":"2026-02-22T20:15:37.034410+0000","last_undegraded":"2026-02-22T20:15:37.034410+0000","last_fullsized":"2026-02-22T20:15:37.034410+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:35:20.222151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035074299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":988,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.033954+0000","last_change":"2026-02-22T19:52:25.383852+0000","last_active":"2026-02-22T20:15:37.033954+0000","last_peered":"2026-02-22T20:15:37.033954+0000","last_clean":"2026-02-22T20:15:37.033954+0000","last_became_active":"2026-02-22T19:52:22.775507+0000","last_became_peered":"2026-02-22T19:52:22.775507+0000","last_unstale":"2026-02-22T20:15:37.033954+0000","last_undegraded":"2026-02-22T20:15:37.033954+0000","last_fullsized":"2026-02-22T20:15:37.033954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:52:30.344481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030984600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":987,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.988007+0000","last_change":"2026-02-22T19:52:24.779287+0000","last_active":"2026-02-22T20:15:37.988007+0000","last_peered":"2026-02-22T20:15:37.988007+0000","last_clean":"2026-02-22T20:15:37.988007+0000","last_became_active":"2026-02-22T19:52:22.774567+0000","last_became_peered":"2026-02-22T19:52:22.774567+0000","last_unstale":"2026-02-22T20:15:37.988007+0000","last_undegraded":"2026-02-22T20:15:37.988007+0000","last_fullsized":"2026-02-22T20:15:37.988007+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:21.765435+0000","last_clean_scrub_stamp":"2026-02-22T19:52:21.765435+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:41:09.322918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036364699999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'72","reported_seq":1077,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.986962+0000","last_change":"2026-02-22T19:52:18.864376+0000","last_active":"2026-02-22T20:15:37.986962+0000","last_peered":"2026-02-22T20:15:37.986962+0000","last_clean":"2026-02-22T20:15:37.986962+0000","last_became_active":"2026-02-22T19:52:18.863570+0000","last_became_peered":"2026-02-22T19:52:18.863570+0000","last_unstale":"2026-02-22T20:15:37.986962+0000","last_undegraded":"2026-02-22T20:15:37.986962+0000","last_fullsized":"2026-02-22T20:15:37.986962+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:52:17.758264+0000","last_clean_scrub_stamp":"2026-02-22T19:52:17.758264+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:09:52.495537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738653,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269720,"kb_used_data":4036,"kb_used_omap":1,"kb_used_meta":265662,"kb_avail":94102120,"statfs":{"total":96636764160,"available":96360570880,"internally_reserved":0,"allocated":4132864,"data_stored":2779839,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":272038342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738651,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314136,"kb_used_data":4036,"kb_used_omap":1,"kb_used_meta":310078,"kb_avail":94057704,"statfs":{"total":96636764160,"available":96315088896,"internally_reserved":0,"allocated":4132864,"data_stored":2779839,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":317520327},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738652,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":277984,"kb_used_data":3976,"kb_used_omap":1,"kb_used_meta":273982,"kb_avail":94093856,"statfs":{"total":96636764160,"available":96352108544,"internally_reserved":0,"allocated":4071424,"data_stored":2189452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280558022},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:15:44.243 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-22T19:52:21.765435+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=21, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=15, tm_sec=42, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:15:44.243 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-22T19:52:21.765435+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=21, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=15, tm_sec=42, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:15:44.243 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-22T19:52:21.765435+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=21, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=15, tm_sec=42, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:15:44.243 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-22T19:52:21.765435+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=21, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=15, tm_sec=42, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:15:44.243 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-22T19:52:21.765435+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=21, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=15, tm_sec=42, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:15:44.244 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-22T19:52:21.765435+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=21, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=15, tm_sec=42, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:15:44.244 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-22T19:52:21.765435+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=21, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=15, tm_sec=42, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:15:44.244 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-22T19:52:21.765435+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=21, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=15, tm_sec=42, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:15:44.244 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-22T19:52:17.758264+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=52, tm_sec=17, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=15, tm_sec=42, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:15:44.244 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-22T20:16:04.245 DEBUG:teuthology.orchestra.run.trial017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-22T20:16:04.398 INFO:teuthology.orchestra.run.trial017.stdout: 2026-02-22T20:16:04.398 INFO:teuthology.orchestra.run.trial017.stderr:dumped all 2026-02-22T20:16:04.408 INFO:teuthology.orchestra.run.trial017.stdout:{"pg_ready":true,"pg_map":{"version":1206,"stamp":"2026-02-22T20:16:02.945710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":861824,"kb_used_data":12048,"kb_used_omap":4,"kb_used_meta":849723,"kb_avail":282253696,"statfs":{"total":289910292480,"available":289027784704,"internally_reserved":0,"allocated":12337152,"data_stored":7749130,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":870116691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001132"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":994,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:48.113694+0000","last_change":"2026-02-22T20:15:48.113708+0000","last_active":"2026-02-22T20:15:48.113694+0000","last_peered":"2026-02-22T20:15:48.113694+0000","last_clean":"2026-02-22T20:15:48.113694+0000","last_became_active":"2026-02-22T19:52:22.776103+0000","last_became_peered":"2026-02-22T19:52:22.776103+0000","last_unstale":"2026-02-22T20:15:48.113694+0000","last_undegraded":"2026-02-22T20:15:48.113694+0000","last_fullsized":"2026-02-22T20:15:48.113694+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:15:48.113631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:15:48.113631+0000","last_clean_scrub_stamp":"2026-02-22T20:15:48.113631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:31:00.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044655200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":994,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:49.073191+0000","last_change":"2026-02-22T20:15:49.073205+0000","last_active":"2026-02-22T20:15:49.073191+0000","last_peered":"2026-02-22T20:15:49.073191+0000","last_clean":"2026-02-22T20:15:49.073191+0000","last_became_active":"2026-02-22T19:52:22.776089+0000","last_became_peered":"2026-02-22T19:52:22.776089+0000","last_unstale":"2026-02-22T20:15:49.073191+0000","last_undegraded":"2026-02-22T20:15:49.073191+0000","last_fullsized":"2026-02-22T20:15:49.073191+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:15:49.073133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:15:49.073133+0000","last_clean_scrub_stamp":"2026-02-22T20:15:49.073133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T21:28:27.079551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051971799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":995,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:47.128778+0000","last_change":"2026-02-22T20:15:47.128796+0000","last_active":"2026-02-22T20:15:47.128778+0000","last_peered":"2026-02-22T20:15:47.128778+0000","last_clean":"2026-02-22T20:15:47.128778+0000","last_became_active":"2026-02-22T19:52:22.775967+0000","last_became_peered":"2026-02-22T19:52:22.775967+0000","last_unstale":"2026-02-22T20:15:47.128778+0000","last_undegraded":"2026-02-22T20:15:47.128778+0000","last_fullsized":"2026-02-22T20:15:47.128778+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:15:47.128710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:15:47.128710+0000","last_clean_scrub_stamp":"2026-02-22T20:15:47.128710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:57:31.166951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060316299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":995,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:45.100519+0000","last_change":"2026-02-22T20:15:45.100546+0000","last_active":"2026-02-22T20:15:45.100519+0000","last_peered":"2026-02-22T20:15:45.100519+0000","last_clean":"2026-02-22T20:15:45.100519+0000","last_became_active":"2026-02-22T19:52:22.775149+0000","last_became_peered":"2026-02-22T19:52:22.775149+0000","last_unstale":"2026-02-22T20:15:45.100519+0000","last_undegraded":"2026-02-22T20:15:45.100519+0000","last_fullsized":"2026-02-22T20:15:45.100519+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:15:45.100451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:15:45.100451+0000","last_clean_scrub_stamp":"2026-02-22T20:15:45.100451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:47:42.269941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033844000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"495'3","reported_seq":999,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:44.214515+0000","last_change":"2026-02-22T20:15:44.214534+0000","last_active":"2026-02-22T20:15:44.214515+0000","last_peered":"2026-02-22T20:15:44.214515+0000","last_clean":"2026-02-22T20:15:44.214515+0000","last_became_active":"2026-02-22T19:52:22.775845+0000","last_became_peered":"2026-02-22T19:52:22.775845+0000","last_unstale":"2026-02-22T20:15:44.214515+0000","last_undegraded":"2026-02-22T20:15:44.214515+0000","last_fullsized":"2026-02-22T20:15:44.214515+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'3","last_scrub_stamp":"2026-02-22T20:15:44.214450+0000","last_deep_scrub":"495'3","last_deep_scrub_stamp":"2026-02-22T20:15:44.214450+0000","last_clean_scrub_stamp":"2026-02-22T20:15:44.214450+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:50:48.249173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033930199999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":997,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:45.729924+0000","last_change":"2026-02-22T20:15:45.729939+0000","last_active":"2026-02-22T20:15:45.729924+0000","last_peered":"2026-02-22T20:15:45.729924+0000","last_clean":"2026-02-22T20:15:45.729924+0000","last_became_active":"2026-02-22T19:52:22.775858+0000","last_became_peered":"2026-02-22T19:52:22.775858+0000","last_unstale":"2026-02-22T20:15:45.729924+0000","last_undegraded":"2026-02-22T20:15:45.729924+0000","last_fullsized":"2026-02-22T20:15:45.729924+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:15:45.729859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:15:45.729859+0000","last_clean_scrub_stamp":"2026-02-22T20:15:45.729859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:03:04.153989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035074299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":997,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:44.774605+0000","last_change":"2026-02-22T20:15:44.774627+0000","last_active":"2026-02-22T20:15:44.774605+0000","last_peered":"2026-02-22T20:15:44.774605+0000","last_clean":"2026-02-22T20:15:44.774605+0000","last_became_active":"2026-02-22T19:52:22.775507+0000","last_became_peered":"2026-02-22T19:52:22.775507+0000","last_unstale":"2026-02-22T20:15:44.774605+0000","last_undegraded":"2026-02-22T20:15:44.774605+0000","last_fullsized":"2026-02-22T20:15:44.774605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:15:44.774517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:15:44.774517+0000","last_clean_scrub_stamp":"2026-02-22T20:15:44.774517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:13:42.730741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030984600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"495'2","reported_seq":997,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:46.085971+0000","last_change":"2026-02-22T20:15:46.086004+0000","last_active":"2026-02-22T20:15:46.085971+0000","last_peered":"2026-02-22T20:15:46.085971+0000","last_clean":"2026-02-22T20:15:46.085971+0000","last_became_active":"2026-02-22T19:52:22.774567+0000","last_became_peered":"2026-02-22T19:52:22.774567+0000","last_unstale":"2026-02-22T20:15:46.085971+0000","last_undegraded":"2026-02-22T20:15:46.085971+0000","last_fullsized":"2026-02-22T20:15:46.085971+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'2","last_scrub_stamp":"2026-02-22T20:15:46.085880+0000","last_deep_scrub":"495'2","last_deep_scrub_stamp":"2026-02-22T20:15:46.085880+0000","last_clean_scrub_stamp":"2026-02-22T20:15:46.085880+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:59:33.566607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036364699999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"495'74","reported_seq":1088,"reported_epoch":495,"state":"active+clean","last_fresh":"2026-02-22T20:15:44.075698+0000","last_change":"2026-02-22T20:15:44.075728+0000","last_active":"2026-02-22T20:15:44.075698+0000","last_peered":"2026-02-22T20:15:44.075698+0000","last_clean":"2026-02-22T20:15:44.075698+0000","last_became_active":"2026-02-22T19:52:18.863570+0000","last_became_peered":"2026-02-22T19:52:18.863570+0000","last_unstale":"2026-02-22T20:15:44.075698+0000","last_undegraded":"2026-02-22T20:15:44.075698+0000","last_fullsized":"2026-02-22T20:15:44.075698+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'74","last_scrub_stamp":"2026-02-22T20:15:44.075625+0000","last_deep_scrub":"495'74","last_deep_scrub_stamp":"2026-02-22T20:15:44.075625+0000","last_clean_scrub_stamp":"2026-02-22T20:15:44.075625+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:06:16.288361+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12908,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738657,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269716,"kb_used_data":4036,"kb_used_omap":1,"kb_used_meta":265662,"kb_avail":94102124,"statfs":{"total":96636764160,"available":96360574976,"internally_reserved":0,"allocated":4132864,"data_stored":2779839,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":272038342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738655,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314132,"kb_used_data":4036,"kb_used_omap":1,"kb_used_meta":310078,"kb_avail":94057708,"statfs":{"total":96636764160,"available":96315092992,"internally_reserved":0,"allocated":4132864,"data_stored":2779839,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":317520327},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738656,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":277976,"kb_used_data":3976,"kb_used_omap":1,"kb_used_meta":273982,"kb_avail":94093864,"statfs":{"total":96636764160,"available":96352116736,"internally_reserved":0,"allocated":4071424,"data_stored":2189452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280558022},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6454,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:16:04.410 DEBUG:teuthology.orchestra.run.trial017:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2026-02-22T20:16:04.564 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-22T20:16:04.565 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-22T20:16:04.565 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2026-02-22T20:16:04.565 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:16:05.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2026-02-22T20:16:10.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2026-02-22T20:16:10.667 INFO:tasks.ceph.cluster2.osd.0:Stopped 2026-02-22T20:16:10.667 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2026-02-22T20:16:10.667 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:16:10.751 INFO:tasks.ceph.cluster2.osd.1:Stopped 2026-02-22T20:16:10.751 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2026-02-22T20:16:10.751 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:16:15.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 is failed for ~0s 2026-02-22T20:16:15.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2026-02-22T20:16:16.853 INFO:tasks.ceph.cluster2.osd.2:Stopped 2026-02-22T20:16:16.853 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-22T20:16:16.854 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2026-02-22T20:16:16.854 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:16:16.906 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2026-02-22T20:16:16.906 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-22T20:16:16.906 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2026-02-22T20:16:16.906 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:16:16.922 INFO:tasks.ceph.cluster2.mon.a:Stopped 2026-02-22T20:16:16.922 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-22T20:16:16.923 DEBUG:teuthology.orchestra.run.trial017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2026-02-22T20:16:16.975 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@trial017.front.sepia.ceph.com 2026-02-22T20:16:16.976 DEBUG:teuthology.orchestra.run.trial017:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2026-02-22T20:16:17.048 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@trial017.front.sepia.ceph.com 2026-02-22T20:16:17.048 DEBUG:teuthology.orchestra.run.trial017:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2026-02-22T20:16:17.112 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@trial017.front.sepia.ceph.com 2026-02-22T20:16:17.112 DEBUG:teuthology.orchestra.run.trial017:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2026-02-22T20:16:17.181 INFO:tasks.ceph:Archiving mon data... 2026-02-22T20:16:17.183 DEBUG:teuthology.misc:Transferring archived files from trial017:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974/data/cluster2.mon.a.tgz 2026-02-22T20:16:17.183 DEBUG:teuthology.orchestra.run.trial017:> mktemp 2026-02-22T20:16:17.224 INFO:teuthology.orchestra.run.trial017.stdout:/tmp/tmp.Byx2dRN2h5 2026-02-22T20:16:17.224 DEBUG:teuthology.orchestra.run.trial017:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.Byx2dRN2h5 2026-02-22T20:16:17.337 DEBUG:teuthology.orchestra.run.trial017:> sudo chmod 0666 /tmp/tmp.Byx2dRN2h5 2026-02-22T20:16:17.390 DEBUG:teuthology.orchestra.remote:trial017:/tmp/tmp.Byx2dRN2h5 is 505KB 2026-02-22T20:16:17.451 DEBUG:teuthology.orchestra.run.trial017:> rm -fr /tmp/tmp.Byx2dRN2h5 2026-02-22T20:16:17.456 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-22T20:16:17.456 DEBUG:teuthology.orchestra.run.trial007:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T20:16:17.459 DEBUG:teuthology.orchestra.run.trial017:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T20:16:17.505 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-22T20:16:17.521 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2026-02-22T20:16:17.521 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T20:16:17.673 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T20:16:17.673 INFO:teuthology.orchestra.run.trial007.stderr:dumped all 2026-02-22T20:16:17.683 INFO:teuthology.orchestra.run.trial007.stdout:{"pg_ready":true,"pg_map":{"version":1328,"stamp":"2026-02-22T20:16:15.844295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":314916,"kb_used_data":14688,"kb_used_omap":4,"kb_used_meta":300155,"kb_avail":282800604,"statfs":{"total":289910292480,"available":289587818496,"internally_reserved":0,"allocated":15040512,"data_stored":9027466,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":307359059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000971"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.100903+0000","last_change":"2026-02-22T19:51:47.650108+0000","last_active":"2026-02-22T20:15:37.100903+0000","last_peered":"2026-02-22T20:15:37.100903+0000","last_clean":"2026-02-22T20:15:37.100903+0000","last_became_active":"2026-02-22T19:51:45.650713+0000","last_became_peered":"2026-02-22T19:51:45.650713+0000","last_unstale":"2026-02-22T20:15:37.100903+0000","last_undegraded":"2026-02-22T20:15:37.100903+0000","last_fullsized":"2026-02-22T20:15:37.100903+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:19:42.205895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000847958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.101563+0000","last_change":"2026-02-22T19:51:47.649914+0000","last_active":"2026-02-22T20:15:37.101563+0000","last_peered":"2026-02-22T20:15:37.101563+0000","last_clean":"2026-02-22T20:15:37.101563+0000","last_became_active":"2026-02-22T19:51:45.650166+0000","last_became_peered":"2026-02-22T19:51:45.650166+0000","last_unstale":"2026-02-22T20:15:37.101563+0000","last_undegraded":"2026-02-22T20:15:37.101563+0000","last_fullsized":"2026-02-22T20:15:37.101563+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:36:23.935956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071726600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.101877+0000","last_change":"2026-02-22T19:51:47.649520+0000","last_active":"2026-02-22T20:15:37.101877+0000","last_peered":"2026-02-22T20:15:37.101877+0000","last_clean":"2026-02-22T20:15:37.101877+0000","last_became_active":"2026-02-22T19:51:45.650286+0000","last_became_peered":"2026-02-22T19:51:45.650286+0000","last_unstale":"2026-02-22T20:15:37.101877+0000","last_undegraded":"2026-02-22T20:15:37.101877+0000","last_fullsized":"2026-02-22T20:15:37.101877+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:58:53.225941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039283599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.098812+0000","last_change":"2026-02-22T19:51:47.649526+0000","last_active":"2026-02-22T20:15:37.098812+0000","last_peered":"2026-02-22T20:15:37.098812+0000","last_clean":"2026-02-22T20:15:37.098812+0000","last_became_active":"2026-02-22T19:51:45.649840+0000","last_became_peered":"2026-02-22T19:51:45.649840+0000","last_unstale":"2026-02-22T20:15:37.098812+0000","last_undegraded":"2026-02-22T20:15:37.098812+0000","last_fullsized":"2026-02-22T20:15:37.098812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:32:49.997988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042606799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":1186,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:36.670953+0000","last_change":"2026-02-22T19:51:47.647395+0000","last_active":"2026-02-22T20:15:36.670953+0000","last_peered":"2026-02-22T20:15:36.670953+0000","last_clean":"2026-02-22T20:15:36.670953+0000","last_became_active":"2026-02-22T19:51:45.650504+0000","last_became_peered":"2026-02-22T19:51:45.650504+0000","last_unstale":"2026-02-22T20:15:36.670953+0000","last_undegraded":"2026-02-22T20:15:36.670953+0000","last_fullsized":"2026-02-22T20:15:36.670953+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:44:15.940372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033513899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1182,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.168408+0000","last_change":"2026-02-22T19:51:48.129796+0000","last_active":"2026-02-22T20:15:37.168408+0000","last_peered":"2026-02-22T20:15:37.168408+0000","last_clean":"2026-02-22T20:15:37.168408+0000","last_became_active":"2026-02-22T19:51:45.650606+0000","last_became_peered":"2026-02-22T19:51:45.650606+0000","last_unstale":"2026-02-22T20:15:37.168408+0000","last_undegraded":"2026-02-22T20:15:37.168408+0000","last_fullsized":"2026-02-22T20:15:37.168408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:23:01.064130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000278965,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.168504+0000","last_change":"2026-02-22T19:51:48.129782+0000","last_active":"2026-02-22T20:15:37.168504+0000","last_peered":"2026-02-22T20:15:37.168504+0000","last_clean":"2026-02-22T20:15:37.168504+0000","last_became_active":"2026-02-22T19:51:45.650544+0000","last_became_peered":"2026-02-22T19:51:45.650544+0000","last_unstale":"2026-02-22T20:15:37.168504+0000","last_undegraded":"2026-02-22T20:15:37.168504+0000","last_fullsized":"2026-02-22T20:15:37.168504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:48:50.952142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027495800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":1184,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.100577+0000","last_change":"2026-02-22T19:51:47.649910+0000","last_active":"2026-02-22T20:15:37.100577+0000","last_peered":"2026-02-22T20:15:37.100577+0000","last_clean":"2026-02-22T20:15:37.100577+0000","last_became_active":"2026-02-22T19:51:45.649866+0000","last_became_peered":"2026-02-22T19:51:45.649866+0000","last_unstale":"2026-02-22T20:15:37.100577+0000","last_undegraded":"2026-02-22T20:15:37.100577+0000","last_fullsized":"2026-02-22T20:15:37.100577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:51:55.975957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071026300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1275,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.099838+0000","last_change":"2026-02-22T19:51:40.106594+0000","last_active":"2026-02-22T20:15:37.099838+0000","last_peered":"2026-02-22T20:15:37.099838+0000","last_clean":"2026-02-22T20:15:37.099838+0000","last_became_active":"2026-02-22T19:51:40.105788+0000","last_became_peered":"2026-02-22T19:51:40.105788+0000","last_unstale":"2026-02-22T20:15:37.099838+0000","last_undegraded":"2026-02-22T20:15:37.099838+0000","last_fullsized":"2026-02-22T20:15:37.099838+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_clean_scrub_stamp":"2026-02-22T19:51:38.632782+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:04:29.975974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705964,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85968,"kb_used_data":4856,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94285872,"statfs":{"total":96636764160,"available":96548732928,"internally_reserved":0,"allocated":4972544,"data_stored":2615564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738665,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112392,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":94259448,"statfs":{"total":96636764160,"available":96521674752,"internally_reserved":0,"allocated":5033984,"data_stored":3205951,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738666,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116556,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":94255284,"statfs":{"total":96636764160,"available":96517410816,"internally_reserved":0,"allocated":5033984,"data_stored":3205951,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":114293191},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:16:17.685 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T20:16:17.835 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T20:16:17.836 INFO:teuthology.orchestra.run.trial007.stderr:dumped all 2026-02-22T20:16:17.846 INFO:teuthology.orchestra.run.trial007.stdout:{"pg_ready":true,"pg_map":{"version":1328,"stamp":"2026-02-22T20:16:15.844295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":314916,"kb_used_data":14688,"kb_used_omap":4,"kb_used_meta":300155,"kb_avail":282800604,"statfs":{"total":289910292480,"available":289587818496,"internally_reserved":0,"allocated":15040512,"data_stored":9027466,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":307359059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000971"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.100903+0000","last_change":"2026-02-22T19:51:47.650108+0000","last_active":"2026-02-22T20:15:37.100903+0000","last_peered":"2026-02-22T20:15:37.100903+0000","last_clean":"2026-02-22T20:15:37.100903+0000","last_became_active":"2026-02-22T19:51:45.650713+0000","last_became_peered":"2026-02-22T19:51:45.650713+0000","last_unstale":"2026-02-22T20:15:37.100903+0000","last_undegraded":"2026-02-22T20:15:37.100903+0000","last_fullsized":"2026-02-22T20:15:37.100903+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:19:42.205895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000847958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.101563+0000","last_change":"2026-02-22T19:51:47.649914+0000","last_active":"2026-02-22T20:15:37.101563+0000","last_peered":"2026-02-22T20:15:37.101563+0000","last_clean":"2026-02-22T20:15:37.101563+0000","last_became_active":"2026-02-22T19:51:45.650166+0000","last_became_peered":"2026-02-22T19:51:45.650166+0000","last_unstale":"2026-02-22T20:15:37.101563+0000","last_undegraded":"2026-02-22T20:15:37.101563+0000","last_fullsized":"2026-02-22T20:15:37.101563+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:36:23.935956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071726600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.101877+0000","last_change":"2026-02-22T19:51:47.649520+0000","last_active":"2026-02-22T20:15:37.101877+0000","last_peered":"2026-02-22T20:15:37.101877+0000","last_clean":"2026-02-22T20:15:37.101877+0000","last_became_active":"2026-02-22T19:51:45.650286+0000","last_became_peered":"2026-02-22T19:51:45.650286+0000","last_unstale":"2026-02-22T20:15:37.101877+0000","last_undegraded":"2026-02-22T20:15:37.101877+0000","last_fullsized":"2026-02-22T20:15:37.101877+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:58:53.225941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039283599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.098812+0000","last_change":"2026-02-22T19:51:47.649526+0000","last_active":"2026-02-22T20:15:37.098812+0000","last_peered":"2026-02-22T20:15:37.098812+0000","last_clean":"2026-02-22T20:15:37.098812+0000","last_became_active":"2026-02-22T19:51:45.649840+0000","last_became_peered":"2026-02-22T19:51:45.649840+0000","last_unstale":"2026-02-22T20:15:37.098812+0000","last_undegraded":"2026-02-22T20:15:37.098812+0000","last_fullsized":"2026-02-22T20:15:37.098812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:32:49.997988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042606799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":1186,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:36.670953+0000","last_change":"2026-02-22T19:51:47.647395+0000","last_active":"2026-02-22T20:15:36.670953+0000","last_peered":"2026-02-22T20:15:36.670953+0000","last_clean":"2026-02-22T20:15:36.670953+0000","last_became_active":"2026-02-22T19:51:45.650504+0000","last_became_peered":"2026-02-22T19:51:45.650504+0000","last_unstale":"2026-02-22T20:15:36.670953+0000","last_undegraded":"2026-02-22T20:15:36.670953+0000","last_fullsized":"2026-02-22T20:15:36.670953+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:44:15.940372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033513899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1182,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.168408+0000","last_change":"2026-02-22T19:51:48.129796+0000","last_active":"2026-02-22T20:15:37.168408+0000","last_peered":"2026-02-22T20:15:37.168408+0000","last_clean":"2026-02-22T20:15:37.168408+0000","last_became_active":"2026-02-22T19:51:45.650606+0000","last_became_peered":"2026-02-22T19:51:45.650606+0000","last_unstale":"2026-02-22T20:15:37.168408+0000","last_undegraded":"2026-02-22T20:15:37.168408+0000","last_fullsized":"2026-02-22T20:15:37.168408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:23:01.064130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000278965,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.168504+0000","last_change":"2026-02-22T19:51:48.129782+0000","last_active":"2026-02-22T20:15:37.168504+0000","last_peered":"2026-02-22T20:15:37.168504+0000","last_clean":"2026-02-22T20:15:37.168504+0000","last_became_active":"2026-02-22T19:51:45.650544+0000","last_became_peered":"2026-02-22T19:51:45.650544+0000","last_unstale":"2026-02-22T20:15:37.168504+0000","last_undegraded":"2026-02-22T20:15:37.168504+0000","last_fullsized":"2026-02-22T20:15:37.168504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:48:50.952142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027495800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":1184,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.100577+0000","last_change":"2026-02-22T19:51:47.649910+0000","last_active":"2026-02-22T20:15:37.100577+0000","last_peered":"2026-02-22T20:15:37.100577+0000","last_clean":"2026-02-22T20:15:37.100577+0000","last_became_active":"2026-02-22T19:51:45.649866+0000","last_became_peered":"2026-02-22T19:51:45.649866+0000","last_unstale":"2026-02-22T20:15:37.100577+0000","last_undegraded":"2026-02-22T20:15:37.100577+0000","last_fullsized":"2026-02-22T20:15:37.100577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:51:55.975957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071026300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1275,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.099838+0000","last_change":"2026-02-22T19:51:40.106594+0000","last_active":"2026-02-22T20:15:37.099838+0000","last_peered":"2026-02-22T20:15:37.099838+0000","last_clean":"2026-02-22T20:15:37.099838+0000","last_became_active":"2026-02-22T19:51:40.105788+0000","last_became_peered":"2026-02-22T19:51:40.105788+0000","last_unstale":"2026-02-22T20:15:37.099838+0000","last_undegraded":"2026-02-22T20:15:37.099838+0000","last_fullsized":"2026-02-22T20:15:37.099838+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_clean_scrub_stamp":"2026-02-22T19:51:38.632782+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:04:29.975974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705964,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85968,"kb_used_data":4856,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94285872,"statfs":{"total":96636764160,"available":96548732928,"internally_reserved":0,"allocated":4972544,"data_stored":2615564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738665,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112392,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":94259448,"statfs":{"total":96636764160,"available":96521674752,"internally_reserved":0,"allocated":5033984,"data_stored":3205951,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738666,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116556,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":94255284,"statfs":{"total":96636764160,"available":96517410816,"internally_reserved":0,"allocated":5033984,"data_stored":3205951,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":114293191},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:16:17.847 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2026-02-22T20:16:17.847 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T20:16:17.989 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T20:16:17.989 INFO:teuthology.orchestra.run.trial007.stderr:dumped all 2026-02-22T20:16:17.998 INFO:teuthology.orchestra.run.trial007.stdout:{"pg_ready":true,"pg_map":{"version":1329,"stamp":"2026-02-22T20:16:17.844453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":314916,"kb_used_data":14688,"kb_used_omap":4,"kb_used_meta":300155,"kb_avail":282800604,"statfs":{"total":289910292480,"available":289587818496,"internally_reserved":0,"allocated":15040512,"data_stored":9027466,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":307359059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000961"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.100903+0000","last_change":"2026-02-22T19:51:47.650108+0000","last_active":"2026-02-22T20:15:37.100903+0000","last_peered":"2026-02-22T20:15:37.100903+0000","last_clean":"2026-02-22T20:15:37.100903+0000","last_became_active":"2026-02-22T19:51:45.650713+0000","last_became_peered":"2026-02-22T19:51:45.650713+0000","last_unstale":"2026-02-22T20:15:37.100903+0000","last_undegraded":"2026-02-22T20:15:37.100903+0000","last_fullsized":"2026-02-22T20:15:37.100903+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:19:42.205895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000847958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.101563+0000","last_change":"2026-02-22T19:51:47.649914+0000","last_active":"2026-02-22T20:15:37.101563+0000","last_peered":"2026-02-22T20:15:37.101563+0000","last_clean":"2026-02-22T20:15:37.101563+0000","last_became_active":"2026-02-22T19:51:45.650166+0000","last_became_peered":"2026-02-22T19:51:45.650166+0000","last_unstale":"2026-02-22T20:15:37.101563+0000","last_undegraded":"2026-02-22T20:15:37.101563+0000","last_fullsized":"2026-02-22T20:15:37.101563+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:36:23.935956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071726600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.101877+0000","last_change":"2026-02-22T19:51:47.649520+0000","last_active":"2026-02-22T20:15:37.101877+0000","last_peered":"2026-02-22T20:15:37.101877+0000","last_clean":"2026-02-22T20:15:37.101877+0000","last_became_active":"2026-02-22T19:51:45.650286+0000","last_became_peered":"2026-02-22T19:51:45.650286+0000","last_unstale":"2026-02-22T20:15:37.101877+0000","last_undegraded":"2026-02-22T20:15:37.101877+0000","last_fullsized":"2026-02-22T20:15:37.101877+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:58:53.225941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039283599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.098812+0000","last_change":"2026-02-22T19:51:47.649526+0000","last_active":"2026-02-22T20:15:37.098812+0000","last_peered":"2026-02-22T20:15:37.098812+0000","last_clean":"2026-02-22T20:15:37.098812+0000","last_became_active":"2026-02-22T19:51:45.649840+0000","last_became_peered":"2026-02-22T19:51:45.649840+0000","last_unstale":"2026-02-22T20:15:37.098812+0000","last_undegraded":"2026-02-22T20:15:37.098812+0000","last_fullsized":"2026-02-22T20:15:37.098812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:32:49.997988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042606799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":1186,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:36.670953+0000","last_change":"2026-02-22T19:51:47.647395+0000","last_active":"2026-02-22T20:15:36.670953+0000","last_peered":"2026-02-22T20:15:36.670953+0000","last_clean":"2026-02-22T20:15:36.670953+0000","last_became_active":"2026-02-22T19:51:45.650504+0000","last_became_peered":"2026-02-22T19:51:45.650504+0000","last_unstale":"2026-02-22T20:15:36.670953+0000","last_undegraded":"2026-02-22T20:15:36.670953+0000","last_fullsized":"2026-02-22T20:15:36.670953+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:44:15.940372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033513899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1182,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.168408+0000","last_change":"2026-02-22T19:51:48.129796+0000","last_active":"2026-02-22T20:15:37.168408+0000","last_peered":"2026-02-22T20:15:37.168408+0000","last_clean":"2026-02-22T20:15:37.168408+0000","last_became_active":"2026-02-22T19:51:45.650606+0000","last_became_peered":"2026-02-22T19:51:45.650606+0000","last_unstale":"2026-02-22T20:15:37.168408+0000","last_undegraded":"2026-02-22T20:15:37.168408+0000","last_fullsized":"2026-02-22T20:15:37.168408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:23:01.064130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000278965,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.168504+0000","last_change":"2026-02-22T19:51:48.129782+0000","last_active":"2026-02-22T20:15:37.168504+0000","last_peered":"2026-02-22T20:15:37.168504+0000","last_clean":"2026-02-22T20:15:37.168504+0000","last_became_active":"2026-02-22T19:51:45.650544+0000","last_became_peered":"2026-02-22T19:51:45.650544+0000","last_unstale":"2026-02-22T20:15:37.168504+0000","last_undegraded":"2026-02-22T20:15:37.168504+0000","last_fullsized":"2026-02-22T20:15:37.168504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:48:50.952142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027495800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":1184,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.100577+0000","last_change":"2026-02-22T19:51:47.649910+0000","last_active":"2026-02-22T20:15:37.100577+0000","last_peered":"2026-02-22T20:15:37.100577+0000","last_clean":"2026-02-22T20:15:37.100577+0000","last_became_active":"2026-02-22T19:51:45.649866+0000","last_became_peered":"2026-02-22T19:51:45.649866+0000","last_unstale":"2026-02-22T20:15:37.100577+0000","last_undegraded":"2026-02-22T20:15:37.100577+0000","last_fullsized":"2026-02-22T20:15:37.100577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:51:55.975957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071026300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1275,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.099838+0000","last_change":"2026-02-22T19:51:40.106594+0000","last_active":"2026-02-22T20:15:37.099838+0000","last_peered":"2026-02-22T20:15:37.099838+0000","last_clean":"2026-02-22T20:15:37.099838+0000","last_became_active":"2026-02-22T19:51:40.105788+0000","last_became_peered":"2026-02-22T19:51:40.105788+0000","last_unstale":"2026-02-22T20:15:37.099838+0000","last_undegraded":"2026-02-22T20:15:37.099838+0000","last_fullsized":"2026-02-22T20:15:37.099838+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_clean_scrub_stamp":"2026-02-22T19:51:38.632782+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:04:29.975974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705964,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85968,"kb_used_data":4856,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94285872,"statfs":{"total":96636764160,"available":96548732928,"internally_reserved":0,"allocated":4972544,"data_stored":2615564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738665,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112392,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":94259448,"statfs":{"total":96636764160,"available":96521674752,"internally_reserved":0,"allocated":5033984,"data_stored":3205951,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738666,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116556,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":94255284,"statfs":{"total":96636764160,"available":96517410816,"internally_reserved":0,"allocated":5033984,"data_stored":3205951,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":114293191},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:16:17.999 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-22T20:16:18.152 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T20:16:18.152 INFO:teuthology.orchestra.run.trial007.stdout:{"epoch":592,"fsid":"a0d80545-8670-4b47-b26d-3047f07e2d2b","created":"2026-02-22T19:51:29.491054+0000","modified":"2026-02-22T20:15:36.664613+0000","last_up_change":"2026-02-22T19:51:38.632782+0000","last_in_change":"2026-02-22T19:51:30.076793+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T19:51:37.655569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T19:51:43.649708+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"07d40a12-e882-4b68-af30-165696dfeff5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6802","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6803","nonce":344374375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6804","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6805","nonce":344374375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6808","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6809","nonce":344374375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6806","nonce":344374375},{"type":"v1","addr":"10.20.193.7:6807","nonce":344374375}]},"public_addr":"10.20.193.7:6803/344374375","cluster_addr":"10.20.193.7:6805/344374375","heartbeat_back_addr":"10.20.193.7:6809/344374375","heartbeat_front_addr":"10.20.193.7:6807/344374375","state":["exists","up"]},{"osd":1,"uuid":"3656b485-906a-4e26-908d-986c907b2604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6810","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6811","nonce":2581020783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6812","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6813","nonce":2581020783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6816","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6817","nonce":2581020783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6814","nonce":2581020783},{"type":"v1","addr":"10.20.193.7:6815","nonce":2581020783}]},"public_addr":"10.20.193.7:6811/2581020783","cluster_addr":"10.20.193.7:6813/2581020783","heartbeat_back_addr":"10.20.193.7:6817/2581020783","heartbeat_front_addr":"10.20.193.7:6815/2581020783","state":["exists","up"]},{"osd":2,"uuid":"2c972c13-b786-41d4-9f1b-ae529fc17749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6818","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6819","nonce":461788605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6820","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6821","nonce":461788605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6824","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6825","nonce":461788605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.7:6822","nonce":461788605},{"type":"v1","addr":"10.20.193.7:6823","nonce":461788605}]},"public_addr":"10.20.193.7:6819/461788605","cluster_addr":"10.20.193.7:6821/461788605","heartbeat_back_addr":"10.20.193.7:6825/461788605","heartbeat_front_addr":"10.20.193.7:6823/461788605","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:36.538925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:36.494710+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T19:51:37.228811+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.17:0/1979219910":"2026-02-22T21:15:14.639077+0000","10.20.193.17:0/3988580775":"2026-02-22T21:15:14.639220+0000","10.20.193.17:0/2986931460":"2026-02-22T21:15:14.637777+0000","10.20.193.17:0/995951461":"2026-02-22T21:12:18.085929+0000","10.20.193.17:0/17592873":"2026-02-22T21:12:48.642412+0000","10.20.193.17:0/2469789236":"2026-02-22T21:12:48.641763+0000","10.20.193.17:0/2519863405":"2026-02-22T21:15:14.639366+0000","10.20.193.17:0/3343515487":"2026-02-22T21:12:48.642017+0000","10.20.193.17:0/117446367":"2026-02-22T21:14:37.870149+0000","10.20.193.17:0/2616230526":"2026-02-22T21:12:48.642219+0000","10.20.193.17:0/1233747191":"2026-02-22T21:12:48.642316+0000","10.20.193.17:0/3740485807":"2026-02-22T21:12:48.642523+0000","10.20.193.17:0/2344085336":"2026-02-22T21:12:17.829223+0000","10.20.193.17:0/3601880843":"2026-02-22T21:14:44.061614+0000","10.20.193.17:0/2608282034":"2026-02-22T21:15:14.638151+0000","10.20.193.17:0/2443902547":"2026-02-22T21:15:14.638291+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":209,"length":1},{"begin":212,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T20:16:19.166 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2026-02-22T20:16:19.166 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:16:19.250 INFO:teuthology.orchestra.run.trial007.stdout:{ 2026-02-22T20:16:19.251 INFO:teuthology.orchestra.run.trial007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T20:16:19.251 INFO:teuthology.orchestra.run.trial007.stdout:} 2026-02-22T20:16:19.260 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2026-02-22T20:16:19.404 INFO:teuthology.orchestra.run.trial007.stderr:instructed osd(s) 0 to deep-scrub 2026-02-22T20:16:19.415 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2026-02-22T20:16:19.415 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:16:19.489 INFO:teuthology.orchestra.run.trial007.stdout:{ 2026-02-22T20:16:19.490 INFO:teuthology.orchestra.run.trial007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:16:19.490 INFO:teuthology.orchestra.run.trial007.stdout:} 2026-02-22T20:16:19.501 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2026-02-22T20:16:19.653 INFO:teuthology.orchestra.run.trial007.stderr:instructed osd(s) 1 to deep-scrub 2026-02-22T20:16:19.664 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2026-02-22T20:16:19.664 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-22T20:16:19.744 INFO:teuthology.orchestra.run.trial007.stdout:{ 2026-02-22T20:16:19.744 INFO:teuthology.orchestra.run.trial007.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:16:19.744 INFO:teuthology.orchestra.run.trial007.stdout:} 2026-02-22T20:16:19.753 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2026-02-22T20:16:19.893 INFO:teuthology.orchestra.run.trial007.stderr:instructed osd(s) 2 to deep-scrub 2026-02-22T20:16:19.904 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T20:16:20.053 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T20:16:20.054 INFO:teuthology.orchestra.run.trial007.stderr:dumped all 2026-02-22T20:16:20.063 INFO:teuthology.orchestra.run.trial007.stdout:{"pg_ready":true,"pg_map":{"version":1330,"stamp":"2026-02-22T20:16:19.844663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":314916,"kb_used_data":14688,"kb_used_omap":4,"kb_used_meta":300155,"kb_avail":282800604,"statfs":{"total":289910292480,"available":289587818496,"internally_reserved":0,"allocated":15040512,"data_stored":9027466,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":307359059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001017"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.100903+0000","last_change":"2026-02-22T19:51:47.650108+0000","last_active":"2026-02-22T20:15:37.100903+0000","last_peered":"2026-02-22T20:15:37.100903+0000","last_clean":"2026-02-22T20:15:37.100903+0000","last_became_active":"2026-02-22T19:51:45.650713+0000","last_became_peered":"2026-02-22T19:51:45.650713+0000","last_unstale":"2026-02-22T20:15:37.100903+0000","last_undegraded":"2026-02-22T20:15:37.100903+0000","last_fullsized":"2026-02-22T20:15:37.100903+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:19:42.205895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000847958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.101563+0000","last_change":"2026-02-22T19:51:47.649914+0000","last_active":"2026-02-22T20:15:37.101563+0000","last_peered":"2026-02-22T20:15:37.101563+0000","last_clean":"2026-02-22T20:15:37.101563+0000","last_became_active":"2026-02-22T19:51:45.650166+0000","last_became_peered":"2026-02-22T19:51:45.650166+0000","last_unstale":"2026-02-22T20:15:37.101563+0000","last_undegraded":"2026-02-22T20:15:37.101563+0000","last_fullsized":"2026-02-22T20:15:37.101563+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:36:23.935956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071726600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.101877+0000","last_change":"2026-02-22T19:51:47.649520+0000","last_active":"2026-02-22T20:15:37.101877+0000","last_peered":"2026-02-22T20:15:37.101877+0000","last_clean":"2026-02-22T20:15:37.101877+0000","last_became_active":"2026-02-22T19:51:45.650286+0000","last_became_peered":"2026-02-22T19:51:45.650286+0000","last_unstale":"2026-02-22T20:15:37.101877+0000","last_undegraded":"2026-02-22T20:15:37.101877+0000","last_fullsized":"2026-02-22T20:15:37.101877+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:58:53.225941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039283599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.098812+0000","last_change":"2026-02-22T19:51:47.649526+0000","last_active":"2026-02-22T20:15:37.098812+0000","last_peered":"2026-02-22T20:15:37.098812+0000","last_clean":"2026-02-22T20:15:37.098812+0000","last_became_active":"2026-02-22T19:51:45.649840+0000","last_became_peered":"2026-02-22T19:51:45.649840+0000","last_unstale":"2026-02-22T20:15:37.098812+0000","last_undegraded":"2026-02-22T20:15:37.098812+0000","last_fullsized":"2026-02-22T20:15:37.098812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:32:49.997988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042606799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":1186,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:36.670953+0000","last_change":"2026-02-22T19:51:47.647395+0000","last_active":"2026-02-22T20:15:36.670953+0000","last_peered":"2026-02-22T20:15:36.670953+0000","last_clean":"2026-02-22T20:15:36.670953+0000","last_became_active":"2026-02-22T19:51:45.650504+0000","last_became_peered":"2026-02-22T19:51:45.650504+0000","last_unstale":"2026-02-22T20:15:36.670953+0000","last_undegraded":"2026-02-22T20:15:36.670953+0000","last_fullsized":"2026-02-22T20:15:36.670953+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:44:15.940372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033513899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1182,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.168408+0000","last_change":"2026-02-22T19:51:48.129796+0000","last_active":"2026-02-22T20:15:37.168408+0000","last_peered":"2026-02-22T20:15:37.168408+0000","last_clean":"2026-02-22T20:15:37.168408+0000","last_became_active":"2026-02-22T19:51:45.650606+0000","last_became_peered":"2026-02-22T19:51:45.650606+0000","last_unstale":"2026-02-22T20:15:37.168408+0000","last_undegraded":"2026-02-22T20:15:37.168408+0000","last_fullsized":"2026-02-22T20:15:37.168408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:23:01.064130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000278965,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1183,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.168504+0000","last_change":"2026-02-22T19:51:48.129782+0000","last_active":"2026-02-22T20:15:37.168504+0000","last_peered":"2026-02-22T20:15:37.168504+0000","last_clean":"2026-02-22T20:15:37.168504+0000","last_became_active":"2026-02-22T19:51:45.650544+0000","last_became_peered":"2026-02-22T19:51:45.650544+0000","last_unstale":"2026-02-22T20:15:37.168504+0000","last_undegraded":"2026-02-22T20:15:37.168504+0000","last_fullsized":"2026-02-22T20:15:37.168504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:48:50.952142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027495800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":1184,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.100577+0000","last_change":"2026-02-22T19:51:47.649910+0000","last_active":"2026-02-22T20:15:37.100577+0000","last_peered":"2026-02-22T20:15:37.100577+0000","last_clean":"2026-02-22T20:15:37.100577+0000","last_became_active":"2026-02-22T19:51:45.649866+0000","last_became_peered":"2026-02-22T19:51:45.649866+0000","last_unstale":"2026-02-22T20:15:37.100577+0000","last_undegraded":"2026-02-22T20:15:37.100577+0000","last_fullsized":"2026-02-22T20:15:37.100577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:44.643552+0000","last_clean_scrub_stamp":"2026-02-22T19:51:44.643552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:51:55.975957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071026300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1275,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:15:37.099838+0000","last_change":"2026-02-22T19:51:40.106594+0000","last_active":"2026-02-22T20:15:37.099838+0000","last_peered":"2026-02-22T20:15:37.099838+0000","last_clean":"2026-02-22T20:15:37.099838+0000","last_became_active":"2026-02-22T19:51:40.105788+0000","last_became_peered":"2026-02-22T19:51:40.105788+0000","last_unstale":"2026-02-22T20:15:37.099838+0000","last_undegraded":"2026-02-22T20:15:37.099838+0000","last_fullsized":"2026-02-22T20:15:37.099838+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T19:51:38.632782+0000","last_clean_scrub_stamp":"2026-02-22T19:51:38.632782+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:04:29.975974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705965,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85968,"kb_used_data":4856,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94285872,"statfs":{"total":96636764160,"available":96548732928,"internally_reserved":0,"allocated":4972544,"data_stored":2615564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112392,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":94259448,"statfs":{"total":96636764160,"available":96521674752,"internally_reserved":0,"allocated":5033984,"data_stored":3205951,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116556,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":94255284,"statfs":{"total":96636764160,"available":96517410816,"internally_reserved":0,"allocated":5033984,"data_stored":3205951,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":114293191},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:16:20.065 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-22T19:51:44.643552+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=44, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=16, tm_sec=18, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:16:20.065 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-22T19:51:44.643552+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=44, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=16, tm_sec=18, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:16:20.065 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-22T19:51:44.643552+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=44, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=16, tm_sec=18, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:16:20.065 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-22T19:51:44.643552+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=44, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=16, tm_sec=18, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:16:20.065 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-22T19:51:44.643552+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=44, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=16, tm_sec=18, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:16:20.065 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-22T19:51:44.643552+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=44, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=16, tm_sec=18, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:16:20.066 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-22T19:51:44.643552+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=44, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=16, tm_sec=18, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:16:20.066 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-22T19:51:44.643552+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=44, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=16, tm_sec=18, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:16:20.066 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-22T19:51:38.632782+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=19, tm_min=51, tm_sec=38, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=20, tm_min=16, tm_sec=18, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T20:16:20.066 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-22T20:16:20.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 has been restored 2026-02-22T20:16:40.067 DEBUG:teuthology.orchestra.run.trial007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-22T20:16:40.207 INFO:teuthology.orchestra.run.trial007.stdout: 2026-02-22T20:16:40.207 INFO:teuthology.orchestra.run.trial007.stderr:dumped all 2026-02-22T20:16:40.216 INFO:teuthology.orchestra.run.trial007.stdout:{"pg_ready":true,"pg_map":{"version":1340,"stamp":"2026-02-22T20:16:39.846402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":314896,"kb_used_data":14688,"kb_used_omap":4,"kb_used_meta":300155,"kb_avail":282800624,"statfs":{"total":289910292480,"available":289587838976,"internally_reserved":0,"allocated":15040512,"data_stored":9027466,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4781,"internal_metadata":307359059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000894"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1192,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:16:25.244775+0000","last_change":"2026-02-22T20:16:25.244802+0000","last_active":"2026-02-22T20:16:25.244775+0000","last_peered":"2026-02-22T20:16:25.244775+0000","last_clean":"2026-02-22T20:16:25.244775+0000","last_became_active":"2026-02-22T19:51:45.650713+0000","last_became_peered":"2026-02-22T19:51:45.650713+0000","last_unstale":"2026-02-22T20:16:25.244775+0000","last_undegraded":"2026-02-22T20:16:25.244775+0000","last_fullsized":"2026-02-22T20:16:25.244775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:16:25.244667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:16:25.244667+0000","last_clean_scrub_stamp":"2026-02-22T20:16:25.244667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:04:17.393193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000847958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1192,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:16:22.225624+0000","last_change":"2026-02-22T20:16:22.225638+0000","last_active":"2026-02-22T20:16:22.225624+0000","last_peered":"2026-02-22T20:16:22.225624+0000","last_clean":"2026-02-22T20:16:22.225624+0000","last_became_active":"2026-02-22T19:51:45.650166+0000","last_became_peered":"2026-02-22T19:51:45.650166+0000","last_unstale":"2026-02-22T20:16:22.225624+0000","last_undegraded":"2026-02-22T20:16:22.225624+0000","last_fullsized":"2026-02-22T20:16:22.225624+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:16:22.225567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:16:22.225567+0000","last_clean_scrub_stamp":"2026-02-22T20:16:22.225567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:29:59.474529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071726600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1192,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:16:23.208975+0000","last_change":"2026-02-22T20:16:23.209001+0000","last_active":"2026-02-22T20:16:23.208975+0000","last_peered":"2026-02-22T20:16:23.208975+0000","last_clean":"2026-02-22T20:16:23.208975+0000","last_became_active":"2026-02-22T19:51:45.650286+0000","last_became_peered":"2026-02-22T19:51:45.650286+0000","last_unstale":"2026-02-22T20:16:23.208975+0000","last_undegraded":"2026-02-22T20:16:23.208975+0000","last_fullsized":"2026-02-22T20:16:23.208975+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:16:23.208867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:16:23.208867+0000","last_clean_scrub_stamp":"2026-02-22T20:16:23.208867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:50:08.464407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039283599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1192,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:16:21.189580+0000","last_change":"2026-02-22T20:16:21.189603+0000","last_active":"2026-02-22T20:16:21.189580+0000","last_peered":"2026-02-22T20:16:21.189580+0000","last_clean":"2026-02-22T20:16:21.189580+0000","last_became_active":"2026-02-22T19:51:45.649840+0000","last_became_peered":"2026-02-22T19:51:45.649840+0000","last_unstale":"2026-02-22T20:16:21.189580+0000","last_undegraded":"2026-02-22T20:16:21.189580+0000","last_fullsized":"2026-02-22T20:16:21.189580+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:16:21.189496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:16:21.189496+0000","last_clean_scrub_stamp":"2026-02-22T20:16:21.189496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:46:58.351795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042606799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"592'3","reported_seq":1196,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:16:19.461228+0000","last_change":"2026-02-22T20:16:19.461247+0000","last_active":"2026-02-22T20:16:19.461228+0000","last_peered":"2026-02-22T20:16:19.461228+0000","last_clean":"2026-02-22T20:16:19.461228+0000","last_became_active":"2026-02-22T19:51:45.650504+0000","last_became_peered":"2026-02-22T19:51:45.650504+0000","last_unstale":"2026-02-22T20:16:19.461228+0000","last_undegraded":"2026-02-22T20:16:19.461228+0000","last_fullsized":"2026-02-22T20:16:19.461228+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'3","last_scrub_stamp":"2026-02-22T20:16:19.461187+0000","last_deep_scrub":"592'3","last_deep_scrub_stamp":"2026-02-22T20:16:19.461187+0000","last_clean_scrub_stamp":"2026-02-22T20:16:19.461187+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:28:51.237304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033513899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1191,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:16:21.530021+0000","last_change":"2026-02-22T20:16:21.530037+0000","last_active":"2026-02-22T20:16:21.530021+0000","last_peered":"2026-02-22T20:16:21.530021+0000","last_clean":"2026-02-22T20:16:21.530021+0000","last_became_active":"2026-02-22T19:51:45.650606+0000","last_became_peered":"2026-02-22T19:51:45.650606+0000","last_unstale":"2026-02-22T20:16:21.530021+0000","last_undegraded":"2026-02-22T20:16:21.530021+0000","last_fullsized":"2026-02-22T20:16:21.530021+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:16:21.529956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:16:21.529956+0000","last_clean_scrub_stamp":"2026-02-22T20:16:21.529956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:32:16.656894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000278965,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1192,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:16:20.480639+0000","last_change":"2026-02-22T20:16:20.480662+0000","last_active":"2026-02-22T20:16:20.480639+0000","last_peered":"2026-02-22T20:16:20.480639+0000","last_clean":"2026-02-22T20:16:20.480639+0000","last_became_active":"2026-02-22T19:51:45.650544+0000","last_became_peered":"2026-02-22T19:51:45.650544+0000","last_unstale":"2026-02-22T20:16:20.480639+0000","last_undegraded":"2026-02-22T20:16:20.480639+0000","last_fullsized":"2026-02-22T20:16:20.480639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T20:16:20.480521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T20:16:20.480521+0000","last_clean_scrub_stamp":"2026-02-22T20:16:20.480521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:29:10.672423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027495800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"592'2","reported_seq":1194,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:16:24.230535+0000","last_change":"2026-02-22T20:16:24.230566+0000","last_active":"2026-02-22T20:16:24.230535+0000","last_peered":"2026-02-22T20:16:24.230535+0000","last_clean":"2026-02-22T20:16:24.230535+0000","last_became_active":"2026-02-22T19:51:45.649866+0000","last_became_peered":"2026-02-22T19:51:45.649866+0000","last_unstale":"2026-02-22T20:16:24.230535+0000","last_undegraded":"2026-02-22T20:16:24.230535+0000","last_fullsized":"2026-02-22T20:16:24.230535+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'2","last_scrub_stamp":"2026-02-22T20:16:24.230459+0000","last_deep_scrub":"592'2","last_deep_scrub_stamp":"2026-02-22T20:16:24.230459+0000","last_clean_scrub_stamp":"2026-02-22T20:16:24.230459+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:57:00.975172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071026300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"592'74","reported_seq":1286,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-22T20:16:20.223574+0000","last_change":"2026-02-22T20:16:20.223602+0000","last_active":"2026-02-22T20:16:20.223574+0000","last_peered":"2026-02-22T20:16:20.223574+0000","last_clean":"2026-02-22T20:16:20.223574+0000","last_became_active":"2026-02-22T19:51:40.105788+0000","last_became_peered":"2026-02-22T19:51:40.105788+0000","last_unstale":"2026-02-22T20:16:20.223574+0000","last_undegraded":"2026-02-22T20:16:20.223574+0000","last_fullsized":"2026-02-22T20:16:20.223574+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'74","last_scrub_stamp":"2026-02-22T20:16:20.223518+0000","last_deep_scrub":"592'74","last_deep_scrub_stamp":"2026-02-22T20:16:20.223518+0000","last_clean_scrub_stamp":"2026-02-22T20:16:20.223518+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T20:50:14.926102+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12890,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705969,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85960,"kb_used_data":4856,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94285880,"statfs":{"total":96636764160,"available":96548741120,"internally_reserved":0,"allocated":4972544,"data_stored":2615564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112388,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":94259452,"statfs":{"total":96636764160,"available":96521678848,"internally_reserved":0,"allocated":5033984,"data_stored":3205951,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738671,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116548,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":94255292,"statfs":{"total":96636764160,"available":96517419008,"internally_reserved":0,"allocated":5033984,"data_stored":3205951,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":114293191},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6445,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T20:16:40.218 DEBUG:teuthology.orchestra.run.trial007:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2026-02-22T20:16:40.375 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-22T20:16:40.375 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-22T20:16:40.376 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2026-02-22T20:16:40.376 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:16:45.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2026-02-22T20:16:46.477 INFO:tasks.ceph.cluster1.osd.0:Stopped 2026-02-22T20:16:46.478 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2026-02-22T20:16:46.478 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:16:46.578 INFO:tasks.ceph.cluster1.osd.1:Stopped 2026-02-22T20:16:46.578 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2026-02-22T20:16:46.578 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:16:46.657 INFO:tasks.ceph.cluster1.osd.2:Stopped 2026-02-22T20:16:46.657 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-22T20:16:46.658 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2026-02-22T20:16:46.658 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:16:46.686 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2026-02-22T20:16:46.686 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-22T20:16:46.686 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2026-02-22T20:16:46.687 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T20:16:46.747 INFO:tasks.ceph.cluster1.mon.a:Stopped 2026-02-22T20:16:46.747 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-22T20:16:46.748 DEBUG:teuthology.orchestra.run.trial007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2026-02-22T20:16:46.802 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@trial007.front.sepia.ceph.com 2026-02-22T20:16:46.803 DEBUG:teuthology.orchestra.run.trial007:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2026-02-22T20:16:46.860 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@trial007.front.sepia.ceph.com 2026-02-22T20:16:46.861 DEBUG:teuthology.orchestra.run.trial007:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2026-02-22T20:16:46.920 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@trial007.front.sepia.ceph.com 2026-02-22T20:16:46.921 DEBUG:teuthology.orchestra.run.trial007:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2026-02-22T20:16:46.976 INFO:tasks.ceph:Archiving mon data... 2026-02-22T20:16:46.977 DEBUG:teuthology.misc:Transferring archived files from trial007:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974/data/cluster1.mon.a.tgz 2026-02-22T20:16:46.977 DEBUG:teuthology.orchestra.run.trial007:> mktemp 2026-02-22T20:16:47.023 INFO:teuthology.orchestra.run.trial007.stdout:/tmp/tmp.7e0TXqGWeY 2026-02-22T20:16:47.024 DEBUG:teuthology.orchestra.run.trial007:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.7e0TXqGWeY 2026-02-22T20:16:47.145 DEBUG:teuthology.orchestra.run.trial007:> sudo chmod 0666 /tmp/tmp.7e0TXqGWeY 2026-02-22T20:16:47.201 DEBUG:teuthology.orchestra.remote:trial007:/tmp/tmp.7e0TXqGWeY is 596KB 2026-02-22T20:16:47.262 DEBUG:teuthology.orchestra.run.trial007:> rm -fr /tmp/tmp.7e0TXqGWeY 2026-02-22T20:16:47.266 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-22T20:16:47.267 DEBUG:teuthology.orchestra.run.trial007:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T20:16:47.309 DEBUG:teuthology.orchestra.run.trial017:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T20:16:47.364 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-22T20:16:47.369 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-22T20:16:47.369 INFO:tasks.ceph:Archiving crash dumps... 2026-02-22T20:16:47.371 DEBUG:teuthology.misc:Transferring archived files from trial007:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974/remote/trial007/crash 2026-02-22T20:16:47.373 DEBUG:teuthology.orchestra.run.trial007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-22T20:16:47.417 DEBUG:teuthology.misc:Transferring archived files from trial017:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974/remote/trial017/crash 2026-02-22T20:16:47.418 DEBUG:teuthology.orchestra.run.trial017:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-22T20:16:47.427 INFO:tasks.ceph:Compressing logs... 2026-02-22T20:16:47.428 DEBUG:teuthology.orchestra.run.trial007:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T20:16:47.465 DEBUG:teuthology.orchestra.run.trial017:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T20:16:47.472 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20826.log 2026-02-22T20:16:47.495 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20852.log 2026-02-22T20:16:47.495 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.20826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19470.log 2026-02-22T20:16:47.495 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20826.log.gz 2026-02-22T20:16:47.495 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.20852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19302.log 2026-02-22T20:16:47.495 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20852.log.gz 2026-02-22T20:16:47.495 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2026-02-22T20:16:47.496 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19470.log.gz 2026-02-22T20:16:47.496 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19302.log.gz 2026-02-22T20:16:47.496 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19164.log 2026-02-22T20:16:47.496 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2026-02-22T20:16:47.496 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19164.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2026-02-22T20:16:47.496 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19164.log.gz 2026-02-22T20:16:47.496 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19301.log 2026-02-22T20:16:47.496 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19454.log 2026-02-22T20:16:47.496 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21008.log 2026-02-22T20:16:47.496 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19301.log.gz 2026-02-22T20:16:47.496 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20930.log 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19454.log.gz 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.21008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19380.log 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21008.log.gz 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.20930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19068.log 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20930.log.gz 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19224.log 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19380.log.gz 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19068.log.gz 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21032.log 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19224.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19224.log.gz 2026-02-22T20:16:47.497 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20878.log 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.21032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19376.log 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21032.log.gz 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.20878.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20878.log.gz 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16289.log 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19376.log.gz 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.12576.log 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.16289.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16289.log.gz 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16196.log 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/ceph.tmp-client.admin.12576.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.12576.log.gz 2026-02-22T20:16:47.498 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20748.log 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.16196.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16196.log.gz 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19190.log 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19554.log 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.20748.log: 93.6% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19190.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20748.log.gz 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19190.log.gz 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19453.log 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19093.log 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19554.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19554.log.gz 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20800.log 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19453.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19453.log.gz 2026-02-22T20:16:47.499 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19093.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19093.log.gz 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20774.log 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.20800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20982.log 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20800.log.gz 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.20774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19223.log 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20774.log.gz 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.20982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19580.log 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20982.log.gz 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr: 86.7% -- replaced with /var/log/ceph/cluster1.log.gz 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16372.log 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19223.log: /var/log/ceph/cluster1-client.admin.19580.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19580.log.gz 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19223.log.gz 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19118.log 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19528.log 2026-02-22T20:16:47.500 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.16372.log: /var/log/ceph/cluster1-client.admin.19118.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16372.log.gz 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr: 80.6% -- replaced with /var/log/ceph/cluster1-client.admin.19118.log.gz 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20904.log 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16437.log 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19528.log.gz 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.20904.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20904.log.gz 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.16437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20956.log 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16437.log.gz 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16313.log 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-osd.0.log: /var/log/ceph/cluster1-client.admin.20956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19375.log 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20956.log.gz 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.16313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19300.log 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16313.log.gz 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19138.log 2026-02-22T20:16:47.501 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19375.log.gz 2026-02-22T20:16:47.502 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19300.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19300.log.gz 2026-02-22T20:16:47.502 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19222.log 2026-02-22T20:16:47.502 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2026-02-22T20:16:47.502 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19138.log.gz 2026-02-22T20:16:47.502 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19222.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19222.log.gz 2026-02-22T20:16:47.502 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19007.log 2026-02-22T20:16:47.502 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16337.log 2026-02-22T20:16:47.502 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.19007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20722.log 2026-02-22T20:16:47.502 INFO:teuthology.orchestra.run.trial007.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19007.log.gz 2026-02-22T20:16:47.502 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.16337.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16337.log.gz 2026-02-22T20:16:47.502 INFO:teuthology.orchestra.run.trial007.stderr:/var/log/ceph/cluster1-client.admin.20722.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20722.log.gz 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52918.log 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27032.log 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48474.log 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52918.log.gz 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.27032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20085.log 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.48474.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50960.log 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48474.log.gz 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.27032.log.gz 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.20085.log: 80.4% -- replaced with /var/log/ceph/cluster1-client.admin.20085.log.gz 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35260.log 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19529.log 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.50960.log: /var/log/ceph/cluster1-client.admin.35260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51190.log 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.admin.50960.log.gz 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19529.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.35260.log.gz 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41921.log 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19529.log.gz 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52193.log 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.51190.log: /var/log/ceph/cluster1-client.mirror.41921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26369.log 2026-02-22T20:16:47.527 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41921.log.gz 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.51190.log.gz 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53052.log 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52193.log.gz 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46862.log 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.26369.log: /var/log/ceph/cluster2-client.mirror.53052.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47611.log 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr: 83.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53052.log.gz 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46862.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.26369.log.gz 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53775.log 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46862.log.gz 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41259.log 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53775.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53775.log.gz 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40479.log 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:gzip 89.0% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.47611.log.gz -- 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster2-client.mirror.35553.log 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.41259.log: /var/log/ceph/cluster1-client.admin.40479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57922.log 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.41259.log.gz 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.40479.log.gz 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.35553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52405.log 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48739.log 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35553.log.gz 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36687.log 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57922.log.gz 2026-02-22T20:16:47.528 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52405.log: /var/log/ceph/cluster1-client.mirror.48739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19955.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48739.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38039.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.36687.log: /var/log/ceph/cluster2-client.admin.19955.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.36687.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.21507.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19955.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.38039.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42262.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.21507.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31871.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.38039.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.42262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27665.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.31871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29855.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.21507.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42262.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31871.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.27665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20204.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.27665.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.29855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52321.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.20204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53235.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.29855.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.20204.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46359.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52321.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45515.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.53235.log: /var/log/ceph/cluster1-client.mirror.46359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41286.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53235.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45515.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46359.log.gz 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33107.log 2026-02-22T20:16:47.529 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35835.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45515.log.gz 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.41286.log: /var/log/ceph/cluster2-client.admin.33107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43556.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.33107.log.gz 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.41286.log.gz 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.35835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52963.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52405.log.gz 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53940.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.35835.log.gz 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32522.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52963.log.gz 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33383.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53940.log: /var/log/ceph/cluster2-client.admin.32522.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43556.log.gz 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44915.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.32522.log.gz 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.33383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34719.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33383.log.gz 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35185.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53940.log.gz 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.34719.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.25343.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.35185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26850.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.34719.log.gz 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.3.25343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20238.log 2026-02-22T20:16:47.530 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.26850.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43187.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44915.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48934.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.35185.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.20238.log: /var/log/ceph/cluster2-client.mirror.43187.log: 58.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20238.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36816.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.48934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32180.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36816.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.48934.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40814.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.36816.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27107.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.32180.log: /var/log/ceph/cluster2-client.mirror.40814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26480.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.27107.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40814.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39850.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22395.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.32180.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.26480.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.27107.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.39850.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20676.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.26480.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.22395.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37752.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39850.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.20676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37496.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr: 55.2% -- replaced with /var/log/ceph/cluster2-client.mirror.20676.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.37752.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.12647.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.22395.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.37752.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20878.log 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.37496.log: /var/log/ceph/ceph.tmp-client.admin.12647.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.12647.log.gz 2026-02-22T20:16:47.531 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45601.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.37496.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.20878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45290.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.20878.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55064.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.26850.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.45290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28007.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45601.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43187.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37257.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: 87.9%/var/log/ceph/cluster1-client.admin.55064.log: -- replaced with /var/log/ceph/cluster1-client.mirror.45290.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.28007.log: 87.8%gzip -- replaced with /var/log/ceph/cluster1-client.admin.28007.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43944.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.55064.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.37257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34404.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.37257.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.50620.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23013.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.34404.log: /var/log/ceph/cluster1-client.mirror.3.50620.log: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.3.50620.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45855.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28488.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23013.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.34404.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45855.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43944.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49380.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45855.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.23013.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19275.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.28488.log: /var/log/ceph/cluster1-client.mirror.49380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43576.log 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49380.log.gz 2026-02-22T20:16:47.532 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56280.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19275.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.43576.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.28488.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.21349.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.56280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36943.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26763.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43576.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.21349.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.56280.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36943.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44962.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.21349.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.36943.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49009.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.26763.log: /var/log/ceph/cluster2-client.mirror.44962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20379.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.26763.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.3.25343.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.49009.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44962.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21168.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.20379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58707.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.20379.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49009.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.21168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46947.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.58707.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41627.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.21168.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58707.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.46947.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46947.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33053.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.41627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32213.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.33053.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.41627.log.gz 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.57413.log 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.32213.log: 87.8%gzip -5 -- replaced with /var/log/ceph/cluster1-client.admin.33053.log.gz --verbose 2026-02-22T20:16:47.533 INFO:teuthology.orchestra.run.trial017.stderr: -- /var/log/ceph/cluster2-client.mirror.44088.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.57413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25691.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23967.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:gzip/var/log/ceph/cluster2-client.admin.25691.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44787.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.32213.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23967.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.25691.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45266.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.23967.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53500.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44787.log: /var/log/ceph/cluster1-client.mirror.45266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19605.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45266.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53500.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53500.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44088.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.22335.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28771.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.19605.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44787.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19605.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.22335.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41439.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.22335.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.28771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46714.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.41439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.59309.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.28771.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.41439.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46714.log: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.2.57413.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47589.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46714.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.59309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24577.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19830.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41893.log 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.24577.log: /var/log/ceph/cluster1-client.admin.19830.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.24577.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19830.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47589.log.gz 2026-02-22T20:16:47.534 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31395.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53259.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.41893.log: /var/log/ceph/cluster2-client.admin.31395.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57513.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.31395.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49725.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.57513.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53259.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43472.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41893.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57513.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52500.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.49725.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44325.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.43472.log: /var/log/ceph/cluster1-client.mirror.52500.log: 87.8% 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43472.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.49725.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52500.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53887.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42993.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.53887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28884.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44325.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.53887.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.42993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40506.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.59309.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.28884.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.28884.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29167.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20467.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.40506.log: /var/log/ceph/cluster2-client.admin.29167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36545.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.40506.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.29167.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.20467.log: 67.2% -- replaced with /var/log/ceph/cluster2-client.admin.20467.log.gz 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47695.log 2026-02-22T20:16:47.535 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.36545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44851.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47695.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.36545.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26504.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19093.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47695.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.26504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40218.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19093.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.admin.19093.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.26504.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53826.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.40218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21148.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29828.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53826.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.21148.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21301.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44851.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42993.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40218.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.21148.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.29828.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.21301.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.29828.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.21301.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36150.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56018.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.36150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19630.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.36150.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.56018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43210.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.51676.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 87.7%/var/log/ceph/cluster1-client.admin.19630.log: -- replaced with /var/log/ceph/cluster2-client.mirror.56018.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19630.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20187.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.51676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.48379.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43210.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.20187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34946.log 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20187.log.gz 2026-02-22T20:16:47.536 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.3.48379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50267.log 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.3.48379.log.gz 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22508.log 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.34946.log: /var/log/ceph/cluster1-client.admin.50267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31529.log 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.34946.log.gz 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.50267.log.gz 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.22508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36051.log 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.31529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34772.log 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.22508.log.gz 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36051.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.31529.log.gz 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55086.log 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.36051.log.gz 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.34772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45919.log 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.55086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.55413.log 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.55086.log.gz 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45919.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.51676.log.gz 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51163.log 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34772.log.gz 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60050.log 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.2.55413.log: /var/log/ceph/cluster1-client.admin.51163.log: 19.0% -- replaced with /var/log/ceph/cluster2-client.mirror.2.55413.log.gz 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45919.log.gz 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29328.log 2026-02-22T20:16:47.537 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.51163.log.gz 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.60050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37523.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.29328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47501.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.37523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28824.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.29328.log.gz 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26024.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.37523.log.gz 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.28824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36970.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47501.log.gz 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.26024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43252.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30019.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36970.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.28824.log.gz 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43252.log: gzip -5 --verbose -- 87.8% /var/log/ceph/cluster1-client.admin.25171.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.36970.log.gz 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43252.log.gz 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54967.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.30019.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.26024.log.gz 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.25171.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39402.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30019.log.gz 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.54967.log: gzip 87.6% -5 -- replaced with /var/log/ceph/cluster1-client.admin.25171.log.gz --verbose 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr: -- /var/log/ceph/cluster2-client.admin.58051.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.39402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20710.log 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.54967.log.gz 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.58051.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.39402.log.gz 2026-02-22T20:16:47.543 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35367.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.58051.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.20710.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52427.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60050.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 56.7% -- replaced with /var/log/ceph/cluster2-client.admin.20710.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.35367.log: --verbose -- /var/log/ceph/cluster1-client.admin.31719.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16427.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.31719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33179.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35367.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.31719.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52896.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.16427.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16427.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.33179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41466.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.52896.log: -- replaced with /var/log/ceph/cluster1-client.admin.33179.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42391.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.41466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37967.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.42391.log: gzip 87.6% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.admin.41466.log.gz -- 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster2-client.mirror.44943.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52896.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.42391.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.37967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24066.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.37967.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.20873.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44943.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.24066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59257.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.3.20873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39429.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.24066.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.59257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47523.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59257.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.39429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43923.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.39429.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41208.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37886.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43923.log: /var/log/ceph/cluster2-client.mirror.41208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45752.log 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47523.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41208.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.37886.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43923.log.gz 2026-02-22T20:16:47.544 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41184.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.37886.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52427.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.20873.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.45752.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32264.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.41184.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45752.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44553.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41184.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.32264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41385.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.32264.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44553.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23166.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44553.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55826.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.41385.log: /var/log/ceph/cluster1-client.admin.23166.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20153.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.41385.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.23166.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.55826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47015.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55826.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.20153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44663.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20153.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.47015.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47015.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38550.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58579.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44663.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.54234.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.38550.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44663.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.58579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38918.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58579.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.3.54234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52261.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.38918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42289.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38550.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52261.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.59300.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 83.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52261.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.42289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25022.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39743.log 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42289.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.59300.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38918.log.gz 2026-02-22T20:16:47.545 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.25022.log: 19.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.59300.log.gz 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24637.log 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.39743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43861.log 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.54234.log.gz 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34297.log 88.6% 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.39743.log.gz 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24637.log: /var/log/ceph/cluster1-client.mirror.43861.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47303.log 87.7% 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.24637.log.gz 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22860.log 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.34297.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43861.log.gz 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47303.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.34297.log.gz 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53662.log 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47303.log.gz 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44046.log 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.22860.log: /var/log/ceph/cluster2-client.mirror.53662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58422.log 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.22860.log.gz 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39351.log 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.58422.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44046.log.gz 2026-02-22T20:16:47.547 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46156.log 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.25022.log.gz 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58422.log.gz 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53662.log.gz 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.39351.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41834.log 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46156.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39351.log.gz 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33331.log 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46156.log.gz 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.41834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19113.log 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.41834.log.gz 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.33331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19043.log 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44423.log 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19113.log.gz 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19043.log: 90.3% 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33331.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.19043.log.gz 2026-02-22T20:16:47.548 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.549 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29561.log 2026-02-22T20:16:47.549 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44423.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60657.log 2026-02-22T20:16:47.549 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.29561.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46419.log 2026-02-22T20:16:47.549 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44423.log.gz 2026-02-22T20:16:47.549 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26342.log 2026-02-22T20:16:47.549 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60657.log.gz 2026-02-22T20:16:47.550 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29561.log.gz 2026-02-22T20:16:47.550 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46419.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46419.log.gz 2026-02-22T20:16:47.550 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30303.log 2026-02-22T20:16:47.550 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.26342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27803.log 2026-02-22T20:16:47.551 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.26342.log.gz 2026-02-22T20:16:47.551 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.30303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53860.log 2026-02-22T20:16:47.551 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.27803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19780.log 2026-02-22T20:16:47.551 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.27803.log.gz 2026-02-22T20:16:47.551 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.53860.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30303.log.gz 2026-02-22T20:16:47.551 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58165.log 2026-02-22T20:16:47.552 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.53860.log.gz 2026-02-22T20:16:47.552 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.19780.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44283.log 2026-02-22T20:16:47.552 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19780.log.gz 2026-02-22T20:16:47.552 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.58165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29968.log 2026-02-22T20:16:47.553 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.21766.log 2026-02-22T20:16:47.553 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.29968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46635.log 2026-02-22T20:16:47.553 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.21766.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44283.log.gz 2026-02-22T20:16:47.554 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31505.log 2026-02-22T20:16:47.554 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.29968.log.gz 2026-02-22T20:16:47.554 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.46635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51050.log 2026-02-22T20:16:47.554 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46635.log.gz 2026-02-22T20:16:47.554 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.31505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52221.log 2026-02-22T20:16:47.555 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.31505.log.gz 2026-02-22T20:16:47.555 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53803.log 2026-02-22T20:16:47.555 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.51050.log: /var/log/ceph/cluster1-client.mirror.52221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50513.log 2026-02-22T20:16:47.555 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52221.log.gz 2026-02-22T20:16:47.555 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.51050.log.gz 2026-02-22T20:16:47.555 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27154.log 2026-02-22T20:16:47.556 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53803.log.gz 2026-02-22T20:16:47.556 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.50513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.25215.log 2026-02-22T20:16:47.556 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.27154.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.50513.log.gz 2026-02-22T20:16:47.556 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36290.log 2026-02-22T20:16:47.556 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.27154.log.gz 2026-02-22T20:16:47.556 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53545.log 2026-02-22T20:16:47.557 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.1.25215.log: 19.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.25215.log.gz 2026-02-22T20:16:47.557 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.36290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57802.log 2026-02-22T20:16:47.557 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58208.log 2026-02-22T20:16:47.557 INFO:teuthology.orchestra.run.trial017.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53545.log.gz 2026-02-22T20:16:47.558 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19730.log 2026-02-22T20:16:47.558 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.58208.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57802.log.gz 2026-02-22T20:16:47.558 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36290.log.gz 2026-02-22T20:16:47.558 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58680.log 2026-02-22T20:16:47.558 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.19730.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58208.log.gz 2026-02-22T20:16:47.558 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48829.log 2026-02-22T20:16:47.558 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19730.log.gz 2026-02-22T20:16:47.559 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.58680.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23040.log 2026-02-22T20:16:47.559 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.48829.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42967.log 2026-02-22T20:16:47.559 INFO:teuthology.orchestra.run.trial017.stderr: 87.7%/var/log/ceph/cluster1-client.admin.23040.log: -- replaced with /var/log/ceph/cluster1-client.mirror.48829.log.gz 2026-02-22T20:16:47.560 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50192.log 2026-02-22T20:16:47.560 INFO:teuthology.orchestra.run.trial017.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58680.log.gz 2026-02-22T20:16:47.560 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.42967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43058.log 2026-02-22T20:16:47.560 INFO:teuthology.orchestra.run.trial017.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.admin.23040.log.gz 2026-02-22T20:16:47.560 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.50192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53920.log 2026-02-22T20:16:47.560 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.admin.42967.log.gz 2026-02-22T20:16:47.561 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50192.log.gz 2026-02-22T20:16:47.561 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.43058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.21771.log 2026-02-22T20:16:47.561 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.53920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38066.log 2026-02-22T20:16:47.561 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44130.log 2026-02-22T20:16:47.562 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.21771.log: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.21771.log.gz 2026-02-22T20:16:47.562 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.38066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57172.log 2026-02-22T20:16:47.562 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.38066.log.gz 2026-02-22T20:16:47.562 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43058.log.gz 2026-02-22T20:16:47.562 INFO:teuthology.orchestra.run.trial017.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53920.log.gz 2026-02-22T20:16:47.562 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.21837.log 2026-02-22T20:16:47.562 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31341.log 2026-02-22T20:16:47.563 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.57172.log: /var/log/ceph/cluster1-client.mirror.2.21837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.50534.log 2026-02-22T20:16:47.563 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.31341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20311.log 2026-02-22T20:16:47.563 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.31341.log.gz 2026-02-22T20:16:47.563 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.50534.log: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.50534.log.gz 2026-02-22T20:16:47.563 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28034.log 2026-02-22T20:16:47.563 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57172.log.gz 2026-02-22T20:16:47.564 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52547.log 2026-02-22T20:16:47.564 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.20311.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44130.log.gz 2026-02-22T20:16:47.564 INFO:teuthology.orchestra.run.trial017.stderr: 57.4% -- replaced with /var/log/ceph/cluster1-client.admin.20311.log.gz 2026-02-22T20:16:47.564 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.28034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49698.log 2026-02-22T20:16:47.564 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.28034.log.gz 2026-02-22T20:16:47.564 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.51625.log 2026-02-22T20:16:47.565 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.49698.log: -- /var/log/ceph/cluster2-client.admin.19655.log 2026-02-22T20:16:47.565 INFO:teuthology.orchestra.run.trial007.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2026-02-22T20:16:47.565 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.51625.log: 20.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.51625.log.gz 2026-02-22T20:16:47.565 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42831.log 2026-02-22T20:16:47.565 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.49698.log.gz 2026-02-22T20:16:47.565 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19655.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52547.log.gz 2026-02-22T20:16:47.565 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19655.log.gz 2026-02-22T20:16:47.565 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27208.log 2026-02-22T20:16:47.566 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29226.log 2026-02-22T20:16:47.566 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.42831.log: /var/log/ceph/cluster2-client.admin.27208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47413.log 2026-02-22T20:16:47.566 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.27208.log.gz 2026-02-22T20:16:47.566 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.29226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56103.log 2026-02-22T20:16:47.566 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42831.log.gz 2026-02-22T20:16:47.567 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38993.log 2026-02-22T20:16:47.567 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37158.log 2026-02-22T20:16:47.567 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.56103.log: /var/log/ceph/cluster1-client.admin.38993.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47413.log.gz 2026-02-22T20:16:47.567 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36996.log 2026-02-22T20:16:47.567 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29226.log.gz 2026-02-22T20:16:47.568 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.38993.log.gz 2026-02-22T20:16:47.568 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.37158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43741.log 2026-02-22T20:16:47.568 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56103.log.gz 2026-02-22T20:16:47.568 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.36996.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.37158.log.gz 2026-02-22T20:16:47.568 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55997.log 2026-02-22T20:16:47.568 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.43741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46219.log 2026-02-22T20:16:47.569 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23400.log 2026-02-22T20:16:47.569 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.55997.log: 91.0% 89.2% -- replaced with /var/log/ceph/cluster1-client.admin.43741.log.gz 2026-02-22T20:16:47.569 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36996.log.gz 2026-02-22T20:16:47.569 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31946.log 2026-02-22T20:16:47.569 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46219.log.gz 2026-02-22T20:16:47.569 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23400.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58395.log 2026-02-22T20:16:47.570 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55997.log.gz 2026-02-22T20:16:47.570 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19139.log/var/log/ceph/cluster2-client.admin.31946.log: 2026-02-22T20:16:47.570 INFO:teuthology.orchestra.run.trial017.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.admin.23400.log.gz 2026-02-22T20:16:47.570 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.31946.log.gz 2026-02-22T20:16:47.570 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41525.log 2026-02-22T20:16:47.570 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.58395.log: /var/log/ceph/cluster2-client.admin.19139.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19139.log.gz 2026-02-22T20:16:47.570 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.55452.log 2026-02-22T20:16:47.570 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.58395.log.gz 2026-02-22T20:16:47.571 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29508.log 2026-02-22T20:16:47.571 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.41525.log: /var/log/ceph/cluster2-client.mirror.2.55452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46883.log 2026-02-22T20:16:47.571 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41525.log.gz 2026-02-22T20:16:47.571 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.29508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.57489.log 2026-02-22T20:16:47.572 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38361.log 2026-02-22T20:16:47.572 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46883.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.29508.log.gz 2026-02-22T20:16:47.572 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.3.57489.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30348.log 2026-02-22T20:16:47.572 INFO:teuthology.orchestra.run.trial017.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46883.log.gz 2026-02-22T20:16:47.572 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.38361.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45962.log 2026-02-22T20:16:47.573 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32951.log 2026-02-22T20:16:47.573 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.30348.log: /var/log/ceph/cluster2-client.mirror.45962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49644.log 2026-02-22T20:16:47.573 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.32951.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38361.log.gz 2026-02-22T20:16:47.573 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58599.log 2026-02-22T20:16:47.573 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39797.log 2026-02-22T20:16:47.574 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.49644.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45962.log.gz 2026-02-22T20:16:47.574 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.58599.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.49644.log.gz 2026-02-22T20:16:47.574 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29140.log 2026-02-22T20:16:47.574 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30348.log.gz 2026-02-22T20:16:47.574 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.39797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36257.log 2026-02-22T20:16:47.574 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.32951.log.gz 2026-02-22T20:16:47.574 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.39797.log.gz 2026-02-22T20:16:47.574 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.29140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20119.log 2026-02-22T20:16:47.574 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.29140.log.gz 2026-02-22T20:16:47.575 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.36257.log: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.55452.log.gz 2026-02-22T20:16:47.575 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44488.log 2026-02-22T20:16:47.575 INFO:teuthology.orchestra.run.trial007.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2026-02-22T20:16:47.575 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49076.log 2026-02-22T20:16:47.575 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.20119.log: /var/log/ceph/cluster2-client.mirror.44488.log: 67.5% -- replaced with /var/log/ceph/cluster1-client.admin.20119.log.gz 2026-02-22T20:16:47.576 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48987.log 2026-02-22T20:16:47.576 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.57489.log.gz 2026-02-22T20:16:47.576 INFO:teuthology.orchestra.run.trial017.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44488.log.gz 2026-02-22T20:16:47.576 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.49076.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36257.log.gz 2026-02-22T20:16:47.576 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26288.log 2026-02-22T20:16:47.576 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.48987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24197.log 2026-02-22T20:16:47.576 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.49076.log.gz 2026-02-22T20:16:47.576 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.26288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49838.log 2026-02-22T20:16:47.577 INFO:teuthology.orchestra.run.trial017.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48987.log.gz 2026-02-22T20:16:47.577 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24197.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.26288.log.gz 2026-02-22T20:16:47.577 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52986.log 2026-02-22T20:16:47.577 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.24197.log.gz 2026-02-22T20:16:47.577 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.49838.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58599.log.gz 2026-02-22T20:16:47.577 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39923.log 2026-02-22T20:16:47.578 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52986.log: 90.4%gzip -- replaced with /var/log/ceph/cluster1-client.admin.49838.log.gz -5 2026-02-22T20:16:47.578 INFO:teuthology.orchestra.run.trial017.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.22556.log 2026-02-22T20:16:47.578 INFO:teuthology.orchestra.run.trial017.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52986.log.gz 2026-02-22T20:16:47.578 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.39923.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23472.log 2026-02-22T20:16:47.578 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.39923.log.gz 2026-02-22T20:16:47.578 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.22556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21487.log 2026-02-22T20:16:47.579 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23472.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.22556.log.gz 2026-02-22T20:16:47.579 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.23472.log.gzgzip 2026-02-22T20:16:47.579 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.34225.log 2026-02-22T20:16:47.579 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.21487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34464.log 2026-02-22T20:16:47.579 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42531.log 2026-02-22T20:16:47.580 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.34225.log: /var/log/ceph/cluster1-client.mirror.34464.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34464.log.gz 2026-02-22T20:16:47.580 INFO:teuthology.orchestra.run.trial017.stderr:gzip 87.3% -5 -- replaced with /var/log/ceph/cluster1-client.admin.34225.log.gz --verbose 2026-02-22T20:16:47.580 INFO:teuthology.orchestra.run.trial017.stderr: -- /var/log/ceph/cluster1-client.admin.32468.log 2026-02-22T20:16:47.580 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.42531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53334.log 2026-02-22T20:16:47.580 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.32468.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.21487.log.gz 2026-02-22T20:16:47.580 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42531.log.gz 2026-02-22T20:16:47.580 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51589.log 2026-02-22T20:16:47.581 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.53334.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.32468.log.gz 2026-02-22T20:16:47.581 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34488.log 2026-02-22T20:16:47.581 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.53334.log.gz 2026-02-22T20:16:47.581 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.51589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35520.log 2026-02-22T20:16:47.581 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.34488.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51589.log.gz 2026-02-22T20:16:47.581 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58442.log 2026-02-22T20:16:47.582 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34488.log.gz 2026-02-22T20:16:47.582 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.35520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21713.log 2026-02-22T20:16:47.582 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.58442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55925.log 2026-02-22T20:16:47.582 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24241.log 2026-02-22T20:16:47.583 INFO:teuthology.orchestra.run.trial017.stderr: 89.1%/var/log/ceph/cluster2-client.mirror.21713.log: -- replaced with /var/log/ceph/cluster2-client.mirror.35520.log.gz 2026-02-22T20:16:47.583 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.55925.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.55925.log.gz 2026-02-22T20:16:47.583 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45815.log 2026-02-22T20:16:47.583 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24241.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.21713.log.gz 2026-02-22T20:16:47.583 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33026.log 2026-02-22T20:16:47.583 INFO:teuthology.orchestra.run.trial017.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24241.log.gz 2026-02-22T20:16:47.583 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45492.log 2026-02-22T20:16:47.584 INFO:teuthology.orchestra.run.trial017.stderr: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45815.log.gz 2026-02-22T20:16:47.584 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52105.log 2026-02-22T20:16:47.584 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.33026.log: /var/log/ceph/cluster1-client.mirror.45492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45145.log 2026-02-22T20:16:47.584 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.33026.log.gz 2026-02-22T20:16:47.584 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52105.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39175.log 2026-02-22T20:16:47.584 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45145.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52105.log.gz 2026-02-22T20:16:47.585 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- 91.9% /var/log/ceph/cluster1-client.mirror.47654.log -- replaced with /var/log/ceph/cluster1-client.mirror.45492.log.gz 2026-02-22T20:16:47.585 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.585 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.39175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31137.log 2026-02-22T20:16:47.585 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.39175.log.gzgzip 2026-02-22T20:16:47.585 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.24147.log 2026-02-22T20:16:47.585 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.47654.log: /var/log/ceph/cluster1-client.mirror.31137.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36392.log 2026-02-22T20:16:47.586 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.31137.log.gz 2026-02-22T20:16:47.586 INFO:teuthology.orchestra.run.trial017.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.47654.log.gz 2026-02-22T20:16:47.586 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.24147.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43794.log 2026-02-22T20:16:47.586 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36392.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.24147.log.gz 2026-02-22T20:16:47.586 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23499.log 2026-02-22T20:16:47.586 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.36392.log.gz 2026-02-22T20:16:47.586 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36317.log 2026-02-22T20:16:47.587 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.43794.log: /var/log/ceph/cluster1-client.admin.23499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26618.log 2026-02-22T20:16:47.587 INFO:teuthology.orchestra.run.trial017.stderr: 95.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58442.log.gz 2026-02-22T20:16:47.587 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.23499.log.gz 2026-02-22T20:16:47.587 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25144.log 2026-02-22T20:16:47.587 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.36317.log: /var/log/ceph/cluster2-client.mirror.26618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33776.log 2026-02-22T20:16:47.587 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36317.log.gz 2026-02-22T20:16:47.587 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.25144.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43794.log.gz 2026-02-22T20:16:47.587 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27381.log 2026-02-22T20:16:47.588 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.25144.log.gz 2026-02-22T20:16:47.588 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.33776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22248.log 2026-02-22T20:16:47.588 INFO:teuthology.orchestra.run.trial017.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.26618.log.gz 2026-02-22T20:16:47.588 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.27381.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42913.log 2026-02-22T20:16:47.588 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.33776.log.gz 2026-02-22T20:16:47.588 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.22248.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.27381.log.gz 2026-02-22T20:16:47.588 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.22248.log.gz 2026-02-22T20:16:47.588 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19503.log 2026-02-22T20:16:47.589 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.42913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46007.log 2026-02-22T20:16:47.589 INFO:teuthology.orchestra.run.trial017.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.21837.log.gz 2026-02-22T20:16:47.589 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40192.log 2026-02-22T20:16:47.589 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19503.log: /var/log/ceph/cluster1-client.mirror.46007.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19503.log.gz 2026-02-22T20:16:47.589 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.42913.log.gz 2026-02-22T20:16:47.589 INFO:teuthology.orchestra.run.trial017.stderr:gzip 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46007.log.gz 2026-02-22T20:16:47.589 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.60071.log 2026-02-22T20:16:47.589 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45145.log.gz 2026-02-22T20:16:47.590 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.40192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.21325.log 2026-02-22T20:16:47.590 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.40192.log.gz 2026-02-22T20:16:47.590 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.60071.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32892.log 2026-02-22T20:16:47.590 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.60071.log.gz 2026-02-22T20:16:47.590 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.21325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25120.log 2026-02-22T20:16:47.590 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.21325.log.gz 2026-02-22T20:16:47.590 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36572.log 2026-02-22T20:16:47.591 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.32892.log: /var/log/ceph/cluster1-client.mirror.25120.log: gzip 87.8% -5 -- replaced with /var/log/ceph/cluster2-client.admin.32892.log.gz --verbose 2026-02-22T20:16:47.591 INFO:teuthology.orchestra.run.trial017.stderr: -- /var/log/ceph/cluster2-client.mirror.45940.log 2026-02-22T20:16:47.591 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.25120.log.gz 2026-02-22T20:16:47.591 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49482.log 2026-02-22T20:16:47.592 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.21889.log 2026-02-22T20:16:47.592 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.36572.log.gz 2026-02-22T20:16:47.592 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45940.log: /var/log/ceph/cluster2-client.admin.49482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38679.log 2026-02-22T20:16:47.592 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38805.log 2026-02-22T20:16:47.592 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.3.21889.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.49482.log.gz 2026-02-22T20:16:47.593 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.38679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.20768.log 2026-02-22T20:16:47.593 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38679.log.gz 2026-02-22T20:16:47.593 INFO:teuthology.orchestra.run.trial017.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58165.log.gz 2026-02-22T20:16:47.593 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.38805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51340.log 2026-02-22T20:16:47.593 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.38805.log.gz 2026-02-22T20:16:47.593 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33461.log 2026-02-22T20:16:47.594 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.20768.log: /var/log/ceph/cluster1-client.mirror.51340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46970.log 2026-02-22T20:16:47.594 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.33461.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46482.log 2026-02-22T20:16:47.594 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.33461.log.gz 2026-02-22T20:16:47.594 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.46970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25839.log 2026-02-22T20:16:47.595 INFO:teuthology.orchestra.run.trial017.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46970.log.gz 2026-02-22T20:16:47.595 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46482.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51340.log.gz 2026-02-22T20:16:47.595 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41780.log 2026-02-22T20:16:47.595 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46482.log.gz 2026-02-22T20:16:47.595 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32495.log 2026-02-22T20:16:47.596 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.25839.log: /var/log/ceph/cluster1-client.admin.41780.log: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.20768.log.gz 2026-02-22T20:16:47.596 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47457.log 2026-02-22T20:16:47.596 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.25839.log.gz 2026-02-22T20:16:47.596 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41780.log.gz 2026-02-22T20:16:47.596 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.32495.log: gzip -5 --verbose 94.1% -- -- replaced with /var/log/ceph/cluster2-client.mirror.45940.log.gz 2026-02-22T20:16:47.596 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster1-client.mirror.20221.log 2026-02-22T20:16:47.596 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.32495.log.gz 2026-02-22T20:16:47.596 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31692.log 2026-02-22T20:16:47.597 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2026-02-22T20:16:47.597 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.20221.log: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.20221.log.gz 2026-02-22T20:16:47.597 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.31692.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47457.log.gz 2026-02-22T20:16:47.597 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52034.log 2026-02-22T20:16:47.597 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.31692.log.gz 2026-02-22T20:16:47.597 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43837.log 2026-02-22T20:16:47.598 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56253.log 2026-02-22T20:16:47.598 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52034.log.gz 2026-02-22T20:16:47.598 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28744.log 2026-02-22T20:16:47.598 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39100.log 2026-02-22T20:16:47.599 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.56253.log: /var/log/ceph/cluster1-client.admin.28744.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.28744.log.gz 2026-02-22T20:16:47.599 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18993.log 2026-02-22T20:16:47.599 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.56253.log.gz 2026-02-22T20:16:47.599 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36599.log 2026-02-22T20:16:47.599 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.39100.log: /var/log/ceph/cluster2-client.admin.18993.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18993.log.gz 2026-02-22T20:16:47.599 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27539.log 2026-02-22T20:16:47.600 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46439.log 2026-02-22T20:16:47.600 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.36599.log.gz 2026-02-22T20:16:47.600 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43837.log.gz 2026-02-22T20:16:47.600 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39100.log.gz 2026-02-22T20:16:47.600 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.27539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52280.log 2026-02-22T20:16:47.601 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47369.log 2026-02-22T20:16:47.601 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46656.log 2026-02-22T20:16:47.601 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52280.log: /var/log/ceph/cluster2-client.mirror.47369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.20731.log 2026-02-22T20:16:47.601 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47369.log.gz 2026-02-22T20:16:47.602 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37338.log 2026-02-22T20:16:47.602 INFO:teuthology.orchestra.run.trial017.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.27539.log.gz 2026-02-22T20:16:47.602 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.46656.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52280.log.gz 2026-02-22T20:16:47.602 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.20731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39950.log 2026-02-22T20:16:47.602 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.20731.log.gz 2026-02-22T20:16:47.602 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.37338.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46656.log.gz 2026-02-22T20:16:47.602 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.37338.log.gz 2026-02-22T20:16:47.602 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59941.log 2026-02-22T20:16:47.603 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.39950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46590.log 2026-02-22T20:16:47.603 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.59941.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.39950.log.gz 2026-02-22T20:16:47.603 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45561.log 2026-02-22T20:16:47.603 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.46590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.21529.log 2026-02-22T20:16:47.604 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46590.log.gz 2026-02-22T20:16:47.604 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45561.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52574.log 2026-02-22T20:16:47.604 INFO:teuthology.orchestra.run.trial017.stderr: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45561.log.gz 2026-02-22T20:16:47.604 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.21529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46820.log 2026-02-22T20:16:47.605 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.21529.log.gz 2026-02-22T20:16:47.605 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46439.log.gz 2026-02-22T20:16:47.605 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41860.log 2026-02-22T20:16:47.605 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52574.log.gz 2026-02-22T20:16:47.605 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60707.log 2026-02-22T20:16:47.605 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46820.log: /var/log/ceph/cluster1-client.mirror.41860.log: 88.4%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.46820.log.gz 2026-02-22T20:16:47.605 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27565.log 2026-02-22T20:16:47.606 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41860.log.gz 2026-02-22T20:16:47.606 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60707.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/cluster2-client.admin.60707.log.gz /var/log/ceph/cluster1-client.admin.39456.log 2026-02-22T20:16:47.606 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.606 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.27565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44195.log 2026-02-22T20:16:47.607 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.39456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57946.log 2026-02-22T20:16:47.607 INFO:teuthology.orchestra.run.trial017.stderr: 87.6%/var/log/ceph/cluster2-client.mirror.44195.log: -- replaced with /var/log/ceph/cluster1-client.admin.39456.log.gz 2026-02-22T20:16:47.607 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.27565.log.gz 2026-02-22T20:16:47.607 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29193.log 2026-02-22T20:16:47.607 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59941.log.gz 2026-02-22T20:16:47.607 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35045.log 2026-02-22T20:16:47.608 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44195.log.gz 2026-02-22T20:16:47.608 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57946.log.gz 2026-02-22T20:16:47.608 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.29193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33956.log 2026-02-22T20:16:47.608 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39148.log 2026-02-22T20:16:47.609 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.35045.log: /var/log/ceph/cluster1-client.admin.33956.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.35045.log.gz 2026-02-22T20:16:47.609 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38410.log 2026-02-22T20:16:47.609 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.33956.log.gz 2026-02-22T20:16:47.609 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29193.log.gz 2026-02-22T20:16:47.609 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.39148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42418.log 2026-02-22T20:16:47.609 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.39148.log.gz 2026-02-22T20:16:47.609 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37913.log 2026-02-22T20:16:47.610 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.38410.log: /var/log/ceph/cluster1-client.admin.42418.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.42418.log.gz 2026-02-22T20:16:47.610 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47886.log 2026-02-22T20:16:47.610 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.38410.log.gz 2026-02-22T20:16:47.610 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.37913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34096.log 2026-02-22T20:16:47.610 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.37913.log.gz 2026-02-22T20:16:47.611 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30946.log 2026-02-22T20:16:47.611 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.34096.log: gzip 87.8% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.34096.log.gz --verbose 2026-02-22T20:16:47.611 INFO:teuthology.orchestra.run.trial017.stderr: -- /var/log/ceph/cluster1-client.mirror.50492.log 2026-02-22T20:16:47.611 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51745.log 2026-02-22T20:16:47.612 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.20773.log 2026-02-22T20:16:47.612 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.50492.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.30946.log.gz 2026-02-22T20:16:47.612 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.51745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20603.log 2026-02-22T20:16:47.612 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50492.log.gz 2026-02-22T20:16:47.612 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51745.log.gz 2026-02-22T20:16:47.612 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.20773.log: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.20773.log.gz 2026-02-22T20:16:47.612 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47391.log 2026-02-22T20:16:47.613 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.20603.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20603.log.gz 2026-02-22T20:16:47.613 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43523.log 2026-02-22T20:16:47.613 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48907.log 2026-02-22T20:16:47.613 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.43523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25348.log 2026-02-22T20:16:47.614 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.43523.log.gz 2026-02-22T20:16:47.614 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.48907.log: gzip -5 --verbose 89.2% -- /var/log/ceph/cluster2-client.mirror.40742.log -- replaced with /var/log/ceph/cluster2-client.mirror.47391.log.gz 2026-02-22T20:16:47.614 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.614 INFO:teuthology.orchestra.run.trial017.stderr:gzip/var/log/ceph/cluster2-client.mirror.25348.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.28798.log 2026-02-22T20:16:47.614 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.48907.log.gz 2026-02-22T20:16:47.615 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.40742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19433.log 2026-02-22T20:16:47.615 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.28798.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40742.log.gz 2026-02-22T20:16:47.615 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51027.log 2026-02-22T20:16:47.615 INFO:teuthology.orchestra.run.trial017.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.25348.log.gz 94.1% 2026-02-22T20:16:47.615 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47886.log.gz 2026-02-22T20:16:47.615 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.28798.log.gz 2026-02-22T20:16:47.615 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23220.log 2026-02-22T20:16:47.616 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19433.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19433.log.gz 2026-02-22T20:16:47.616 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.51027.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35212.log 2026-02-22T20:16:47.616 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.51027.log.gz 2026-02-22T20:16:47.616 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36789.log 2026-02-22T20:16:47.616 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.23220.log.gz 2026-02-22T20:16:47.616 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.35212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.59915.log 2026-02-22T20:16:47.617 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36789.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.35212.log.gz 2026-02-22T20:16:47.617 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29382.log 2026-02-22T20:16:47.617 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.36789.log.gz 2026-02-22T20:16:47.617 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.59915.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49671.log 2026-02-22T20:16:47.617 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.29382.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.59915.log.gz 2026-02-22T20:16:47.618 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41232.log 2026-02-22T20:16:47.618 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.29382.log.gz 2026-02-22T20:16:47.618 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.49671.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49509.log 2026-02-22T20:16:47.618 INFO:teuthology.orchestra.run.trial017.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.admin.49671.log.gz 2026-02-22T20:16:47.618 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.41232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35341.log 2026-02-22T20:16:47.619 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5/var/log/ceph/cluster2-client.admin.49509.log: --verbose -- /var/log/ceph/cluster2-client.mirror.45472.log 2026-02-22T20:16:47.619 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.41232.log.gz 2026-02-22T20:16:47.619 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.35341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52673.log 2026-02-22T20:16:47.619 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% 90.5% -- replaced with /var/log/ceph/cluster2-client.admin.35341.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.49509.log.gz 2026-02-22T20:16:47.619 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.619 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35440.log 2026-02-22T20:16:47.620 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45472.log: /var/log/ceph/cluster1-client.admin.52673.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.52673.log.gz 2026-02-22T20:16:47.620 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24530.log 2026-02-22T20:16:47.620 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.35440.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46756.log 2026-02-22T20:16:47.620 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24530.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.35440.log.gz 2026-02-22T20:16:47.620 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46398.log 2026-02-22T20:16:47.620 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.24530.log.gz 2026-02-22T20:16:47.621 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49220.log 2026-02-22T20:16:47.621 INFO:teuthology.orchestra.run.trial017.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45472.log.gz 2026-02-22T20:16:47.621 INFO:teuthology.orchestra.run.trial017.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.21889.log.gz 2026-02-22T20:16:47.621 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32240.log 2026-02-22T20:16:47.621 INFO:teuthology.orchestra.run.trial017.stderr: 89.3%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.46756.log.gz -5 2026-02-22T20:16:47.621 INFO:teuthology.orchestra.run.trial017.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.34566.log 2026-02-22T20:16:47.622 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.49220.log: /var/log/ceph/cluster1-client.mirror.32240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48853.log 2026-02-22T20:16:47.622 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.32240.log.gz 2026-02-22T20:16:47.622 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46398.log.gz 2026-02-22T20:16:47.622 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.34566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33701.log 2026-02-22T20:16:47.622 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49220.log.gz 2026-02-22T20:16:47.622 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.34566.log.gz 2026-02-22T20:16:47.622 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.48853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54879.log 2026-02-22T20:16:47.623 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.48853.log.gz/var/log/ceph/cluster1-client.mirror.33701.log: 2026-02-22T20:16:47.623 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21692.log 2026-02-22T20:16:47.623 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.54879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47347.log 2026-02-22T20:16:47.623 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.21692.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.54879.log.gz 2026-02-22T20:16:47.624 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31481.log 2026-02-22T20:16:47.624 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.33701.log.gz 2026-02-22T20:16:47.624 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27065.log 2026-02-22T20:16:47.624 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.31481.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.21692.log.gz 2026-02-22T20:16:47.624 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19805.log 2026-02-22T20:16:47.624 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47347.log.gz 2026-02-22T20:16:47.625 INFO:teuthology.orchestra.run.trial017.stderr: 87.7%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.31481.log.gz 2026-02-22T20:16:47.625 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53165.log 2026-02-22T20:16:47.625 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.27065.log: /var/log/ceph/cluster1-client.admin.19805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20552.log 2026-02-22T20:16:47.625 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19805.log.gz 2026-02-22T20:16:47.625 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52084.log 2026-02-22T20:16:47.625 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53165.log.gz 2026-02-22T20:16:47.625 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.20552.log: 58.3% -- replaced with /var/log/ceph/cluster2-client.mirror.20552.log.gz 2026-02-22T20:16:47.626 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43331.log 2026-02-22T20:16:47.626 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43080.log 2026-02-22T20:16:47.626 INFO:teuthology.orchestra.run.trial017.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52084.log.gz 2026-02-22T20:16:47.626 INFO:teuthology.orchestra.run.trial017.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27065.log.gz 2026-02-22T20:16:47.626 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36889.log 2026-02-22T20:16:47.626 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43331.log.gz 2026-02-22T20:16:47.627 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43080.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32918.log 2026-02-22T20:16:47.627 INFO:teuthology.orchestra.run.trial017.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43080.log.gz 2026-02-22T20:16:47.627 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.36889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39555.log 2026-02-22T20:16:47.627 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.32918.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.36889.log.gz 2026-02-22T20:16:47.627 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52361.log 2026-02-22T20:16:47.628 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.39555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27638.log 2026-02-22T20:16:47.628 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.39555.log.gz 2026-02-22T20:16:47.628 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54932.log 2026-02-22T20:16:47.628 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.32918.log.gz 2026-02-22T20:16:47.629 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49157.log 2026-02-22T20:16:47.629 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52361.log.gz 2026-02-22T20:16:47.629 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.27638.log: /var/log/ceph/cluster1-client.mirror.54932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27779.log 2026-02-22T20:16:47.629 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.27638.log.gz 2026-02-22T20:16:47.629 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.49157.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49157.log.gz 2026-02-22T20:16:47.629 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.25210.log 2026-02-22T20:16:47.629 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33080.log 2026-02-22T20:16:47.630 INFO:teuthology.orchestra.run.trial017.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54932.log.gz 2026-02-22T20:16:47.630 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.27779.log: /var/log/ceph/cluster2-client.mirror.0.25210.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.27779.log.gz 2026-02-22T20:16:47.630 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55950.log 2026-02-22T20:16:47.630 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.33080.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30685.log 2026-02-22T20:16:47.631 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.55950.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.33080.log.gz 2026-02-22T20:16:47.631 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57326.log 2026-02-22T20:16:47.631 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.55950.log.gz 2026-02-22T20:16:47.631 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.30685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60631.log 2026-02-22T20:16:47.631 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.54153.log 2026-02-22T20:16:47.632 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57326.log.gz 2026-02-22T20:16:47.632 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24120.log 2026-02-22T20:16:47.632 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% 89.3% -- replaced with /var/log/ceph/cluster2-client.admin.60631.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.30685.log.gz 2026-02-22T20:16:47.632 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.632 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43167.log 2026-02-22T20:16:47.632 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.54153.log: 19.0% -- replaced with /var/log/ceph/cluster1-client.mirror.2.54153.log.gz 2026-02-22T20:16:47.633 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.24120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41969.log 2026-02-22T20:16:47.633 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.24120.log.gz 2026-02-22T20:16:47.633 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47237.log 2026-02-22T20:16:47.633 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43167.log.gz 2026-02-22T20:16:47.633 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44641.log 2026-02-22T20:16:47.633 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.41969.log: /var/log/ceph/cluster2-client.mirror.47237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57826.log 2026-02-22T20:16:47.634 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44641.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.41969.log.gz 2026-02-22T20:16:47.634 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30605.log 2026-02-22T20:16:47.634 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47237.log.gz 2026-02-22T20:16:47.634 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57826.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41707.log 2026-02-22T20:16:47.634 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57826.log.gz 2026-02-22T20:16:47.635 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44641.log.gz 2026-02-22T20:16:47.635 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22941.log 2026-02-22T20:16:47.635 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.41707.log: 87.7%gzip -- replaced with /var/log/ceph/cluster1-client.admin.30605.log.gz -5 2026-02-22T20:16:47.635 INFO:teuthology.orchestra.run.trial017.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.41807.log 2026-02-22T20:16:47.635 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.22941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.59337.log 2026-02-22T20:16:47.636 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.41807.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41707.log.gz 2026-02-22T20:16:47.636 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48009.log 2026-02-22T20:16:47.636 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.22941.log.gz 2026-02-22T20:16:47.636 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.59337.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.41807.log.gz 2026-02-22T20:16:47.636 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32658.log 2026-02-22T20:16:47.637 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.48009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28986.log 2026-02-22T20:16:47.637 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48009.log.gz 2026-02-22T20:16:47.637 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.32658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40560.log 2026-02-22T20:16:47.637 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.32658.log.gz 2026-02-22T20:16:47.637 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58105.log 2026-02-22T20:16:47.637 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.28986.log: /var/log/ceph/cluster2-client.admin.40560.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.59337.log.gz 2026-02-22T20:16:47.638 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32369.log 2026-02-22T20:16:47.638 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.28986.log.gz 2026-02-22T20:16:47.638 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.40560.log.gz 2026-02-22T20:16:47.638 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.58105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40787.log 2026-02-22T20:16:47.638 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.58105.log.gz 2026-02-22T20:16:47.638 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.32369.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53092.log 2026-02-22T20:16:47.639 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.32369.log.gz 2026-02-22T20:16:47.639 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.40787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19855.log 2026-02-22T20:16:47.639 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.53092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30632.log 2026-02-22T20:16:47.640 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53092.log.gz 2026-02-22T20:16:47.640 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49925.log 2026-02-22T20:16:47.640 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19855.log.gz 2026-02-22T20:16:47.640 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.30632.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40787.log.gz 2026-02-22T20:16:47.640 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24554.log 2026-02-22T20:16:47.640 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.30632.log.gz 2026-02-22T20:16:47.640 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.49925.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37811.log 2026-02-22T20:16:47.641 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.24554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43882.log 2026-02-22T20:16:47.641 INFO:teuthology.orchestra.run.trial017.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.24554.log.gz 2026-02-22T20:16:47.641 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.37811.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46840.log 2026-02-22T20:16:47.642 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.43882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27261.log 2026-02-22T20:16:47.642 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46840.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49925.log.gz 2026-02-22T20:16:47.642 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31027.log 2026-02-22T20:16:47.642 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.27261.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37811.log.gz 2026-02-22T20:16:47.642 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- 90.0% /var/log/ceph/cluster1-client.mirror.58755.log -- replaced with /var/log/ceph/cluster1-client.mirror.43882.log.gz 2026-02-22T20:16:47.643 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.643 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.31027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27357.log 2026-02-22T20:16:47.643 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.31027.log.gz 2026-02-22T20:16:47.643 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.58755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25790.log 2026-02-22T20:16:47.643 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27261.log.gz 2026-02-22T20:16:47.644 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58755.log.gz 2026-02-22T20:16:47.644 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.27357.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46840.log.gz 2026-02-22T20:16:47.644 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31314.log 2026-02-22T20:16:47.644 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.27357.log.gz 2026-02-22T20:16:47.644 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.25790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30375.log 2026-02-22T20:16:47.644 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.31314.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35655.log 2026-02-22T20:16:47.645 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.30375.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.31314.log.gz 2026-02-22T20:16:47.645 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28087.log 2026-02-22T20:16:47.645 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.35655.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30375.log.gz 2026-02-22T20:16:47.645 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22636.log 2026-02-22T20:16:47.645 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.35655.log.gz 2026-02-22T20:16:47.645 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.28087.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33286.log 2026-02-22T20:16:47.646 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54752.log 2026-02-22T20:16:47.646 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.22636.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.28087.log.gz 2026-02-22T20:16:47.646 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.33286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58779.log 2026-02-22T20:16:47.646 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.54752.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54752.log.gz 2026-02-22T20:16:47.646 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47802.log 2026-02-22T20:16:47.647 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.58779.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19354.log 2026-02-22T20:16:47.647 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47802.log: 89.2% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58779.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.33286.log.gz 2026-02-22T20:16:47.647 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.647 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36024.log 2026-02-22T20:16:47.647 INFO:teuthology.orchestra.run.trial017.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.22636.log.gz 2026-02-22T20:16:47.647 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.admin.19354.log: /var/log/ceph/cluster2-client.mirror.52449.log 2026-02-22T20:16:47.648 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19354.log.gz 2026-02-22T20:16:47.648 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.36024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32634.log 2026-02-22T20:16:47.648 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.36024.log.gz 2026-02-22T20:16:47.648 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33667.log 2026-02-22T20:16:47.648 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52449.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47802.log.gz 2026-02-22T20:16:47.648 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.32634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42491.log 2026-02-22T20:16:47.648 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.32634.log.gz 2026-02-22T20:16:47.649 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.33667.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25744.log 2026-02-22T20:16:47.649 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52449.log.gz 2026-02-22T20:16:47.649 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.42491.log: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42491.log.gz 2026-02-22T20:16:47.649 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35709.log 2026-02-22T20:16:47.650 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25917.log 2026-02-22T20:16:47.650 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.25744.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.33667.log.gz 2026-02-22T20:16:47.650 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.35709.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.25790.log.gz 2026-02-22T20:16:47.650 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27459.log 2026-02-22T20:16:47.650 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.35709.log.gz 2026-02-22T20:16:47.650 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.25744.log.gz 2026-02-22T20:16:47.650 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.25917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20635.log 2026-02-22T20:16:47.650 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.25917.log.gz 2026-02-22T20:16:47.651 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.27459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28515.log 2026-02-22T20:16:47.651 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.27459.log.gz 2026-02-22T20:16:47.651 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49103.log 2026-02-22T20:16:47.651 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.20635.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20635.log.gz 2026-02-22T20:16:47.651 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.28515.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.28515.log.gz 2026-02-22T20:16:47.651 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36762.log 2026-02-22T20:16:47.652 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38093.log 2026-02-22T20:16:47.652 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.49103.log: /var/log/ceph/cluster1-client.admin.36762.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.36762.log.gz 2026-02-22T20:16:47.652 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- 87.6% /var/log/ceph/cluster1-client.mirror.55183.log 2026-02-22T20:16:47.652 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.49103.log.gz 2026-02-22T20:16:47.652 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.38093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37603.log 2026-02-22T20:16:47.653 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.38093.log.gz 2026-02-22T20:16:47.653 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.55183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52010.log 2026-02-22T20:16:47.653 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.37603.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55183.log.gz 2026-02-22T20:16:47.653 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59834.log 2026-02-22T20:16:47.654 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23067.log 2026-02-22T20:16:47.654 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52010.log.gz 2026-02-22T20:16:47.654 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.59834.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37603.log.gz 2026-02-22T20:16:47.654 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35413.log 2026-02-22T20:16:47.654 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22761.log 2026-02-22T20:16:47.655 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.59834.log.gz 2026-02-22T20:16:47.655 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.23067.log.gz 2026-02-22T20:16:47.655 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.35413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30973.log 2026-02-22T20:16:47.655 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.22761.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.35413.log.gz 2026-02-22T20:16:47.655 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40715.log 2026-02-22T20:16:47.655 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.22761.log.gz 2026-02-22T20:16:47.655 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54570.log 2026-02-22T20:16:47.656 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.30973.log.gz 2026-02-22T20:16:47.656 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.40715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36518.log 2026-02-22T20:16:47.656 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.40715.log.gz 2026-02-22T20:16:47.656 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.54570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.25303.log 2026-02-22T20:16:47.656 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.36518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20328.log 2026-02-22T20:16:47.656 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54570.log.gz 2026-02-22T20:16:47.657 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.3.25303.log: 19.6% -- replaced with /var/log/ceph/cluster2-client.mirror.3.25303.log.gzgzip 2026-02-22T20:16:47.657 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21673.log 2026-02-22T20:16:47.657 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.36518.log.gz 2026-02-22T20:16:47.657 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.20328.log: 55.2% -- replaced with /var/log/ceph/cluster1-client.mirror.20328.log.gz 2026-02-22T20:16:47.657 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45314.log 2026-02-22T20:16:47.657 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.21673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36078.log 2026-02-22T20:16:47.658 INFO:teuthology.orchestra.run.trial017.stderr: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.21673.log.gz 2026-02-22T20:16:47.658 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.45314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35949.log 2026-02-22T20:16:47.658 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36078.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.45314.log.gz 2026-02-22T20:16:47.658 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31161.log 2026-02-22T20:16:47.658 INFO:teuthology.orchestra.run.trial017.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.36078.log.gz 2026-02-22T20:16:47.658 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.35949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.57390.log 2026-02-22T20:16:47.659 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.31161.log: gzip 87.7% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.35949.log.gz --verbose -- 2026-02-22T20:16:47.659 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster2-client.mirror.48265.log 2026-02-22T20:16:47.659 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.31161.log.gz 2026-02-22T20:16:47.659 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.57390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23633.log 2026-02-22T20:16:47.659 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.57390.log.gz 2026-02-22T20:16:47.660 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45124.log 2026-02-22T20:16:47.660 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.48265.log: /var/log/ceph/cluster1-client.admin.23633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39977.log 2026-02-22T20:16:47.660 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45124.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.23633.log.gz 2026-02-22T20:16:47.660 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44508.log 2026-02-22T20:16:47.661 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45124.log.gz 2026-02-22T20:16:47.661 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.39977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52727.log 2026-02-22T20:16:47.661 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47060.log 2026-02-22T20:16:47.661 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.39977.log.gz 2026-02-22T20:16:47.661 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.52727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24615.log 2026-02-22T20:16:47.662 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.47060.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.52727.log.gz 2026-02-22T20:16:47.662 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36472.log 2026-02-22T20:16:47.662 INFO:teuthology.orchestra.run.trial017.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47060.log.gz 2026-02-22T20:16:47.662 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41654.log 2026-02-22T20:16:47.662 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.36472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46131.log 2026-02-22T20:16:47.663 INFO:teuthology.orchestra.run.trial017.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44508.log.gz 2026-02-22T20:16:47.663 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36472.log.gz 2026-02-22T20:16:47.663 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.41654.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.24615.log.gz 2026-02-22T20:16:47.663 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57970.log 2026-02-22T20:16:47.663 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42510.log 2026-02-22T20:16:47.663 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46131.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.41654.log.gz 2026-02-22T20:16:47.663 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57970.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57970.log.gzgzip 2026-02-22T20:16:47.663 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.60143.log 2026-02-22T20:16:47.664 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58132.log 2026-02-22T20:16:47.664 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.42510.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48265.log.gz 2026-02-22T20:16:47.664 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.54118.log 2026-02-22T20:16:47.664 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60143.log.gz 2026-02-22T20:16:47.664 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.58132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22481.log 2026-02-22T20:16:47.664 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.58132.log.gz 2026-02-22T20:16:47.665 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.54118.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.21766.log.gz 2026-02-22T20:16:47.665 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42510.log.gzgzip 2026-02-22T20:16:47.665 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27306.log 2026-02-22T20:16:47.665 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.22481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60475.log 2026-02-22T20:16:47.665 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.27306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60007.log 2026-02-22T20:16:47.665 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.22481.log.gz 2026-02-22T20:16:47.666 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54615.log 2026-02-22T20:16:47.666 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60475.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60475.log.gz 2026-02-22T20:16:47.666 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.60007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29013.log 2026-02-22T20:16:47.666 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40063.log 2026-02-22T20:16:47.667 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.54615.log: /var/log/ceph/cluster1-client.admin.29013.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54615.log.gz 2026-02-22T20:16:47.667 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.29013.log.gz 2026-02-22T20:16:47.667 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26046.log 2026-02-22T20:16:47.667 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.48323.log 2026-02-22T20:16:47.667 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.40063.log: /var/log/ceph/cluster2-client.mirror.26046.log: gzip -5 --verbose -- 87.8% /var/log/ceph/cluster1-client.mirror.48787.log 2026-02-22T20:16:47.667 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40063.log.gz 2026-02-22T20:16:47.668 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46131.log.gz 2026-02-22T20:16:47.668 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.48323.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.26046.log.gz 2026-02-22T20:16:47.668 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27085.log 2026-02-22T20:16:47.668 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.48323.log.gz 2026-02-22T20:16:47.668 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20287.log 2026-02-22T20:16:47.668 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.48787.log: /var/log/ceph/cluster1-client.mirror.27085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53283.log 2026-02-22T20:16:47.668 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48787.log.gz 2026-02-22T20:16:47.669 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.20287.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.20287.log.gz 2026-02-22T20:16:47.669 INFO:teuthology.orchestra.run.trial017.stderr: 96.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60007.log.gz 2026-02-22T20:16:47.669 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34892.log 2026-02-22T20:16:47.669 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41339.log 2026-02-22T20:16:47.669 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53283.log.gz 2026-02-22T20:16:47.669 INFO:teuthology.orchestra.run.trial017.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.27085.log.gz 2026-02-22T20:16:47.670 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.34892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44704.log 2026-02-22T20:16:47.670 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.34892.log.gz 2026-02-22T20:16:47.670 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.41339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42700.log 2026-02-22T20:16:47.670 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28061.log 2026-02-22T20:16:47.670 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48084.log 2026-02-22T20:16:47.671 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.42700.log: 90.0%/var/log/ceph/cluster2-client.admin.28061.log: -- replaced with /var/log/ceph/cluster1-client.mirror.41339.log.gz 2026-02-22T20:16:47.671 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42700.log.gz 2026-02-22T20:16:47.671 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.28061.log.gz 2026-02-22T20:16:47.671 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41313.log 2026-02-22T20:16:47.671 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.48084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54676.log 2026-02-22T20:16:47.671 INFO:teuthology.orchestra.run.trial017.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44704.log.gz 2026-02-22T20:16:47.671 INFO:teuthology.orchestra.run.trial017.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.27306.log.gz 2026-02-22T20:16:47.672 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.41313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56178.log 2026-02-22T20:16:47.672 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.48084.log.gz 2026-02-22T20:16:47.672 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.54676.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.41313.log.gz 2026-02-22T20:16:47.672 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27235.log 2026-02-22T20:16:47.672 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.54676.log.gz 2026-02-22T20:16:47.672 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.56178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44110.log 2026-02-22T20:16:47.673 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.27235.log: -- /var/log/ceph/cluster2-client.admin.20255.log 2026-02-22T20:16:47.673 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56178.log.gz 2026-02-22T20:16:47.673 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44110.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.27235.log.gz 2026-02-22T20:16:47.673 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52941.log 2026-02-22T20:16:47.673 INFO:teuthology.orchestra.run.trial017.stderr:gzip 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44110.log.gz 2026-02-22T20:16:47.674 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.20255.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20501.log 2026-02-22T20:16:47.674 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52941.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.20255.log.gz 2026-02-22T20:16:47.674 INFO:teuthology.orchestra.run.trial017.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52941.log.gz 2026-02-22T20:16:47.674 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45795.log 2026-02-22T20:16:47.674 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.20501.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32610.log 2026-02-22T20:16:47.674 INFO:teuthology.orchestra.run.trial017.stderr: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.20501.log.gz 2026-02-22T20:16:47.675 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.45795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32441.log 2026-02-22T20:16:47.675 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.32610.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32610.log.gz 2026-02-22T20:16:47.675 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34692.log 2026-02-22T20:16:47.675 INFO:teuthology.orchestra.run.trial017.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45795.log.gz 2026-02-22T20:16:47.675 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.32441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44620.log 2026-02-22T20:16:47.676 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.32441.log.gz 2026-02-22T20:16:47.676 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.34692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43690.log 2026-02-22T20:16:47.676 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.34692.log.gz 2026-02-22T20:16:47.676 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28375.log 2026-02-22T20:16:47.676 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.43690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29253.log 2026-02-22T20:16:47.677 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.28375.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43127.log 2026-02-22T20:16:47.677 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44620.log.gz 2026-02-22T20:16:47.677 INFO:teuthology.orchestra.run.trial017.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43690.log.gz 2026-02-22T20:16:47.677 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.29253.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.29253.log.gz 2026-02-22T20:16:47.677 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.28375.log.gz 2026-02-22T20:16:47.677 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29039.log 2026-02-22T20:16:47.678 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40686.log 2026-02-22T20:16:47.678 INFO:teuthology.orchestra.run.trial017.stderr: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43127.log.gz 2026-02-22T20:16:47.678 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49201.log 2026-02-22T20:16:47.678 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.29039.log: /var/log/ceph/cluster2-client.admin.40686.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.40686.log.gz 2026-02-22T20:16:47.678 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60423.log 2026-02-22T20:16:47.679 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.49201.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25718.log 2026-02-22T20:16:47.679 INFO:teuthology.orchestra.run.trial017.stderr: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49201.log.gz 2026-02-22T20:16:47.679 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29039.log.gz 2026-02-22T20:16:47.679 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31215.log 2026-02-22T20:16:47.679 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60423.log.gz 2026-02-22T20:16:47.679 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.25718.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31188.log 2026-02-22T20:16:47.680 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.25718.log.gz 2026-02-22T20:16:47.680 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.31215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40404.log 2026-02-22T20:16:47.680 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.31215.log.gz 2026-02-22T20:16:47.680 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.31188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19755.log 2026-02-22T20:16:47.680 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.31188.log.gz 2026-02-22T20:16:47.681 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.40404.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33514.log 2026-02-22T20:16:47.681 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.19755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.48374.log 2026-02-22T20:16:47.681 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40404.log.gz 2026-02-22T20:16:47.681 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19755.log.gz 2026-02-22T20:16:47.681 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26816.log 2026-02-22T20:16:47.682 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.33514.log: /var/log/ceph/cluster1-client.mirror.2.48374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38464.log 2026-02-22T20:16:47.682 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.26816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.55512.log 2026-02-22T20:16:47.682 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33514.log.gz 2026-02-22T20:16:47.682 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.38464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55978.log 2026-02-22T20:16:47.682 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.26816.log.gz 2026-02-22T20:16:47.683 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.38464.log.gz 2026-02-22T20:16:47.683 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.3.55512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55162.log 2026-02-22T20:16:47.683 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.55978.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48111.log 2026-02-22T20:16:47.683 INFO:teuthology.orchestra.run.trial017.stderr: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55978.log.gz 2026-02-22T20:16:47.684 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.55162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24467.log 2026-02-22T20:16:47.684 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55162.log.gz 2026-02-22T20:16:47.684 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.48111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.21757.log 2026-02-22T20:16:47.684 INFO:teuthology.orchestra.run.trial017.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.48374.log.gz 2026-02-22T20:16:47.684 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24467.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.48111.log.gz 2026-02-22T20:16:47.684 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26638.log 2026-02-22T20:16:47.685 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.21757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21733.log 2026-02-22T20:16:47.685 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.21757.log.gz 2026-02-22T20:16:47.685 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55876.log 2026-02-22T20:16:47.685 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.26638.log: /var/log/ceph/cluster2-client.mirror.21733.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.24467.log.gz 2026-02-22T20:16:47.685 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54772.log 2026-02-22T20:16:47.686 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.55876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39020.log 2026-02-22T20:16:47.686 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55876.log.gz 2026-02-22T20:16:47.686 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.54772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25611.log 2026-02-22T20:16:47.687 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.39020.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.26638.log.gz 2026-02-22T20:16:47.687 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48057.log 2026-02-22T20:16:47.687 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.39020.log.gz 2026-02-22T20:16:47.687 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.25611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58027.log 2026-02-22T20:16:47.687 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40371.log 2026-02-22T20:16:47.687 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.48057.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.25611.log.gz 2026-02-22T20:16:47.687 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.58027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46693.log 2026-02-22T20:16:47.688 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58027.log.gz 2026-02-22T20:16:47.688 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.40371.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.48057.log.gz 2026-02-22T20:16:47.688 INFO:teuthology.orchestra.run.trial017.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.55512.log.gz 2026-02-22T20:16:47.688 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54772.log.gz 2026-02-22T20:16:47.688 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43774.log 2026-02-22T20:16:47.688 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40431.log 2026-02-22T20:16:47.688 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40371.log.gz 2026-02-22T20:16:47.689 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24836.log 2026-02-22T20:16:47.689 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.40431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38254.log 2026-02-22T20:16:47.689 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46693.log.gz 2026-02-22T20:16:47.689 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40431.log.gz 2026-02-22T20:16:47.689 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.24836.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.24836.log.gz 2026-02-22T20:16:47.689 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50141.log 2026-02-22T20:16:47.690 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.38254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30718.log 2026-02-22T20:16:47.690 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.50141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42940.log 2026-02-22T20:16:47.690 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.38254.log.gz 2026-02-22T20:16:47.690 INFO:teuthology.orchestra.run.trial017.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43774.log.gz 2026-02-22T20:16:47.690 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.30718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40943.log 2026-02-22T20:16:47.691 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50141.log.gz 2026-02-22T20:16:47.691 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.42940.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.30718.log.gz 2026-02-22T20:16:47.691 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49579.log 2026-02-22T20:16:47.691 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.40943.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.21733.log.gz 2026-02-22T20:16:47.691 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.42940.log.gz 2026-02-22T20:16:47.691 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33728.log 2026-02-22T20:16:47.691 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.49579.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40943.log.gz 2026-02-22T20:16:47.692 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51103.log 2026-02-22T20:16:47.692 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49579.log.gz 2026-02-22T20:16:47.692 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.51736.log 2026-02-22T20:16:47.692 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.33728.log: /var/log/ceph/cluster1-client.mirror.51103.log: gzip -5 --verbose -- 87.8% /var/log/ceph/cluster1-client.mirror.28455.log 2026-02-22T20:16:47.692 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.33728.log.gz 2026-02-22T20:16:47.693 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.3.51736.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51103.log.gzgzip 2026-02-22T20:16:47.693 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.23319.log 2026-02-22T20:16:47.693 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.28455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.20745.log 2026-02-22T20:16:47.693 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32739.log 2026-02-22T20:16:47.693 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.28455.log.gz 2026-02-22T20:16:47.694 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.23319.log.gz/var/log/ceph/cluster1-client.mirror.1.20745.log: 2026-02-22T20:16:47.694 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37838.log 2026-02-22T20:16:47.694 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.20745.log.gz 2026-02-22T20:16:47.694 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.32739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36341.log 2026-02-22T20:16:47.694 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.32739.log.gz 2026-02-22T20:16:47.694 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.37838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19199.log 2026-02-22T20:16:47.695 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37838.log.gz 2026-02-22T20:16:47.695 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.36341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48763.log 2026-02-22T20:16:47.695 INFO:teuthology.orchestra.run.trial017.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.3.51736.log.gz 2026-02-22T20:16:47.695 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19199.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36341.log.gz 2026-02-22T20:16:47.695 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28539.log 2026-02-22T20:16:47.695 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19199.log.gz 2026-02-22T20:16:47.695 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.48763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41600.log 2026-02-22T20:16:47.696 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48763.log.gz 2026-02-22T20:16:47.696 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.28539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32342.log 2026-02-22T20:16:47.696 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28539.log.gz 2026-02-22T20:16:47.696 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.41600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60095.log 2026-02-22T20:16:47.697 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.41600.log.gz 2026-02-22T20:16:47.697 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.32342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50072.log 2026-02-22T20:16:47.697 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60095.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60095.log.gz 2026-02-22T20:16:47.697 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27692.log 2026-02-22T20:16:47.697 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.32342.log.gz 2026-02-22T20:16:47.697 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.50072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43614.log 2026-02-22T20:16:47.698 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.27692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30150.log 2026-02-22T20:16:47.698 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43614.log: 87.8%gzip -- replaced with /var/log/ceph/cluster2-client.admin.27692.log.gz -5 2026-02-22T20:16:47.698 INFO:teuthology.orchestra.run.trial017.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.55326.log 2026-02-22T20:16:47.699 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.30150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58884.log 2026-02-22T20:16:47.699 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.55326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52476.log 2026-02-22T20:16:47.699 INFO:teuthology.orchestra.run.trial017.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43614.log.gz 2026-02-22T20:16:47.699 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.58884.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55326.log.gz 2026-02-22T20:16:47.699 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- 89.9% /var/log/ceph/cluster2-client.admin.32127.log 2026-02-22T20:16:47.699 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.30150.log.gz 2026-02-22T20:16:47.700 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.58884.log.gz 2026-02-22T20:16:47.700 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32073.log 2026-02-22T20:16:47.700 INFO:teuthology.orchestra.run.trial017.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52476.log.gz 2026-02-22T20:16:47.700 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.32127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58266.log 2026-02-22T20:16:47.700 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.32127.log.gz 2026-02-22T20:16:47.700 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.32073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34593.log 2026-02-22T20:16:47.701 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.58266.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.32073.log.gz 2026-02-22T20:16:47.701 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55299.log 2026-02-22T20:16:47.701 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.34593.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.58266.log.gz 2026-02-22T20:16:47.701 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35236.log 2026-02-22T20:16:47.701 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.34593.log.gz 2026-02-22T20:16:47.701 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.55299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26872.log 2026-02-22T20:16:47.701 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.55299.log.gz 2026-02-22T20:16:47.701 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.35236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.55380.log 2026-02-22T20:16:47.702 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35236.log.gz 2026-02-22T20:16:47.702 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.26872.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44831.log 2026-02-22T20:16:47.702 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.0.55380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26315.log 2026-02-22T20:16:47.702 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38179.log 2026-02-22T20:16:47.703 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.26872.log.gz 2026-02-22T20:16:47.703 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.26315.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.21427.log 2026-02-22T20:16:47.703 INFO:teuthology.orchestra.run.trial017.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44831.log.gz 2026-02-22T20:16:47.703 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.26315.log.gz 2026-02-22T20:16:47.703 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.38179.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50072.log.gz 2026-02-22T20:16:47.703 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37104.log 2026-02-22T20:16:47.704 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.21427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52170.log 2026-02-22T20:16:47.704 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.37104.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.21427.log.gz 2026-02-22T20:16:47.704 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41552.log 2026-02-22T20:16:47.704 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.37104.log.gz 2026-02-22T20:16:47.704 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30070.log 2026-02-22T20:16:47.704 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52170.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.38179.log.gz/var/log/ceph/cluster2-client.mirror.41552.log: 2026-02-22T20:16:47.704 INFO:teuthology.orchestra.run.trial017.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52170.log.gz 2026-02-22T20:16:47.705 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41552.log.gz 2026-02-22T20:16:47.705 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55135.log 2026-02-22T20:16:47.705 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45443.log 2026-02-22T20:16:47.705 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30070.log: /var/log/ceph/cluster1-client.admin.55135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24972.log 2026-02-22T20:16:47.705 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.30070.log.gz 2026-02-22T20:16:47.706 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.55135.log.gz 2026-02-22T20:16:47.706 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.45443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34539.log 2026-02-22T20:16:47.706 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34844.log 2026-02-22T20:16:47.706 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.24972.log.gz 2026-02-22T20:16:47.706 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.34539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47632.log 2026-02-22T20:16:47.707 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.34844.log: gzip 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.34539.log.gz -5 2026-02-22T20:16:47.707 INFO:teuthology.orchestra.run.trial017.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.34120.log 2026-02-22T20:16:47.707 INFO:teuthology.orchestra.run.trial017.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45443.log.gz 2026-02-22T20:16:47.707 INFO:teuthology.orchestra.run.trial017.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.47632.log: -- replaced with /var/log/ceph/cluster2-client.mirror.34844.log.gz 2026-02-22T20:16:47.707 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31773.log 2026-02-22T20:16:47.708 INFO:teuthology.orchestra.run.trial017.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47632.log.gz 2026-02-22T20:16:47.708 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.34120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31973.log 2026-02-22T20:16:47.708 INFO:teuthology.orchestra.run.trial017.stderr: 87.7%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.34120.log.gz 2026-02-22T20:16:47.708 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.25238.log 2026-02-22T20:16:47.708 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.31773.log: /var/log/ceph/cluster1-client.admin.31973.log: 87.7%gzip -- replaced with /var/log/ceph/cluster2-client.admin.31773.log.gz -5 2026-02-22T20:16:47.708 INFO:teuthology.orchestra.run.trial017.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.0.50543.log 2026-02-22T20:16:47.708 INFO:teuthology.orchestra.run.trial017.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.31973.log.gz 2026-02-22T20:16:47.709 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.1.25238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37029.log 2026-02-22T20:16:47.709 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.50543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33641.log 2026-02-22T20:16:47.709 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.37029.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58932.log 2026-02-22T20:16:47.710 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46905.log 2026-02-22T20:16:47.710 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.33641.log: /var/log/ceph/cluster2-client.mirror.58932.log: gzip 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.33641.log.gz -5 2026-02-22T20:16:47.710 INFO:teuthology.orchestra.run.trial017.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.55346.log 2026-02-22T20:16:47.710 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46905.log: gzip -5 90.3% --verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.37029.log.gz 2026-02-22T20:16:47.710 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster2-client.mirror.46568.log 2026-02-22T20:16:47.711 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46905.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.58932.log.gz 2026-02-22T20:16:47.711 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.711 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.55346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.22362.log 2026-02-22T20:16:47.711 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46568.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36231.log 2026-02-22T20:16:47.712 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46568.log.gz 2026-02-22T20:16:47.712 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.22362.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34919.log 2026-02-22T20:16:47.712 INFO:teuthology.orchestra.run.trial017.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.50543.log.gz 2026-02-22T20:16:47.712 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36231.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.admin.22362.log.gz 2026-02-22T20:16:47.712 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19580.log 2026-02-22T20:16:47.712 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.34919.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43307.log 2026-02-22T20:16:47.712 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.36231.log.gz 2026-02-22T20:16:47.712 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.34919.log.gz 2026-02-22T20:16:47.713 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.19580.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19580.log.gz 2026-02-22T20:16:47.713 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55346.log.gz 2026-02-22T20:16:47.713 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30277.log 2026-02-22T20:16:47.713 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46293.log 2026-02-22T20:16:47.714 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.30277.log: gzip 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43307.log.gz 2026-02-22T20:16:47.714 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49968.log 2026-02-22T20:16:47.714 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.46293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34512.log 2026-02-22T20:16:47.714 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.30277.log.gz 2026-02-22T20:16:47.714 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.49968.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46293.log.gz 2026-02-22T20:16:47.714 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39047.log 2026-02-22T20:16:47.714 INFO:teuthology.orchestra.run.trial017.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.49968.log.gz 2026-02-22T20:16:47.715 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.34512.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50448.log 2026-02-22T20:16:47.715 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.39047.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33260.log 2026-02-22T20:16:47.715 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.34512.log.gz 2026-02-22T20:16:47.715 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.39047.log.gz 2026-02-22T20:16:47.715 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.50448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39229.log 2026-02-22T20:16:47.716 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.33260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52057.log 2026-02-22T20:16:47.716 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.33260.log.gz 2026-02-22T20:16:47.716 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.39229.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53367.log 2026-02-22T20:16:47.716 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.39229.log.gz 2026-02-22T20:16:47.716 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41576.log 2026-02-22T20:16:47.717 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52057.log: /var/log/ceph/cluster2-client.mirror.53367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34171.log 2026-02-22T20:16:47.717 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52057.log.gz 2026-02-22T20:16:47.717 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.41576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35287.log 2026-02-22T20:16:47.717 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41576.log.gz 2026-02-22T20:16:47.718 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.34171.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45707.log 2026-02-22T20:16:47.718 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.35287.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.34171.log.gz 2026-02-22T20:16:47.718 INFO:teuthology.orchestra.run.trial017.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53367.log.gz 2026-02-22T20:16:47.718 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.50674.log 2026-02-22T20:16:47.718 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.35287.log.gz 2026-02-22T20:16:47.718 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45707.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58078.log 2026-02-22T20:16:47.719 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.3.50674.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45707.log.gz 2026-02-22T20:16:47.719 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2026-02-22T20:16:47.719 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19277.log 2026-02-22T20:16:47.719 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.58078.log: /var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23913.log 2026-02-22T20:16:47.719 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19277.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.58078.log.gz 2026-02-22T20:16:47.719 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19277.log.gz 2026-02-22T20:16:47.720 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28402.log 2026-02-22T20:16:47.720 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35862.log 2026-02-22T20:16:47.720 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23913.log: /var/log/ceph/cluster2-client.admin.28402.log: 87.6%gzip -- replaced with /var/log/ceph/cluster1-client.admin.23913.log.gz -5 2026-02-22T20:16:47.720 INFO:teuthology.orchestra.run.trial017.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.30477.log 2026-02-22T20:16:47.721 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.28402.log.gz 2026-02-22T20:16:47.721 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.35862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57194.log 2026-02-22T20:16:47.721 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30477.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.50674.log.gz 2026-02-22T20:16:47.721 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.35862.log.gz 2026-02-22T20:16:47.721 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2026-02-22T20:16:47.721 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50448.log.gz 2026-02-22T20:16:47.721 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.30477.log.gz 2026-02-22T20:16:47.722 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.57194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42149.log 2026-02-22T20:16:47.722 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57194.log.gz 2026-02-22T20:16:47.722 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44809.log 2026-02-22T20:16:47.722 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.42149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39300.log 2026-02-22T20:16:47.723 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44809.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42149.log.gz 2026-02-22T20:16:47.723 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53726.log 2026-02-22T20:16:47.723 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27486.log 2026-02-22T20:16:47.723 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44809.log.gz 2026-02-22T20:16:47.723 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.39300.log: /var/log/ceph/cluster1-client.mirror.53726.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20345.log 2026-02-22T20:16:47.723 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53726.log.gz 2026-02-22T20:16:47.724 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.27486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26978.log 2026-02-22T20:16:47.724 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.27486.log.gz 2026-02-22T20:16:47.724 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.20345.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20944.log 2026-02-22T20:16:47.724 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39300.log.gz 2026-02-22T20:16:47.724 INFO:teuthology.orchestra.run.trial017.stderr: 61.4% -- replaced with /var/log/ceph/cluster1-client.admin.20345.log.gz 2026-02-22T20:16:47.724 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.admin.26978.log: /var/log/ceph/cluster2-client.admin.36204.log 2026-02-22T20:16:47.725 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.20944.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.26978.log.gz 2026-02-22T20:16:47.725 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19880.log 2026-02-22T20:16:47.725 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.20944.log.gz 2026-02-22T20:16:47.725 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35467.log 2026-02-22T20:16:47.725 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.36204.log.gz 2026-02-22T20:16:47.726 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45194.log 2026-02-22T20:16:47.726 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19880.log.gz 2026-02-22T20:16:47.726 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.35467.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.35467.log.gz 2026-02-22T20:16:47.726 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.25201.log 2026-02-22T20:16:47.726 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33233.log 2026-02-22T20:16:47.727 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45194.log.gz 2026-02-22T20:16:47.727 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.0.25201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23689.log 2026-02-22T20:16:47.727 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.25201.log.gz 2026-02-22T20:16:47.727 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.33233.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26107.log 2026-02-22T20:16:47.727 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.33233.log.gz 2026-02-22T20:16:47.727 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24861.log 2026-02-22T20:16:47.728 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.26107.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.23689.log.gz 2026-02-22T20:16:47.728 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49988.log 2026-02-22T20:16:47.728 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24861.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.26107.log.gz 2026-02-22T20:16:47.728 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24861.log.gz 2026-02-22T20:16:47.728 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26429.log 2026-02-22T20:16:47.729 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.49988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29454.log 2026-02-22T20:16:47.729 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.26429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44596.log 2026-02-22T20:16:47.729 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.29454.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.29454.log.gz 2026-02-22T20:16:47.729 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19165.log 2026-02-22T20:16:47.730 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57778.log 2026-02-22T20:16:47.730 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44596.log: /var/log/ceph/cluster2-client.admin.19165.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19165.log.gz 2026-02-22T20:16:47.730 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33434.log 2026-02-22T20:16:47.730 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44596.log.gz 2026-02-22T20:16:47.730 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26429.log.gz 2026-02-22T20:16:47.730 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24786.log 2026-02-22T20:16:47.730 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49988.log.gz 2026-02-22T20:16:47.731 INFO:teuthology.orchestra.run.trial017.stderr: 87.7%/var/log/ceph/cluster2-client.admin.33434.log: -- replaced with /var/log/ceph/cluster1-client.mirror.57778.log.gz 2026-02-22T20:16:47.731 INFO:teuthology.orchestra.run.trial017.stderr:gzip 93.3% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.0.25210.log.gz --verbose 2026-02-22T20:16:47.731 INFO:teuthology.orchestra.run.trial017.stderr: -- /var/log/ceph/cluster2-client.mirror.46798.log 2026-02-22T20:16:47.731 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.33434.log.gz 2026-02-22T20:16:47.731 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.24786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33359.log 2026-02-22T20:16:47.731 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.24786.log.gz 2026-02-22T20:16:47.731 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28171.log 2026-02-22T20:16:47.732 INFO:teuthology.orchestra.run.trial017.stderr: 89.2%/var/log/ceph/cluster2-client.mirror.33359.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46798.log.gz 2026-02-22T20:16:47.732 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.20833.log 2026-02-22T20:16:47.732 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33359.log.gz 2026-02-22T20:16:47.732 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31610.log 2026-02-22T20:16:47.732 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.28171.log: /var/log/ceph/cluster1-client.mirror.3.20833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25866.log 19.6% 2026-02-22T20:16:47.732 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.3.20833.log.gz 2026-02-22T20:16:47.733 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.28171.log.gz 2026-02-22T20:16:47.733 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.31610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19930.log 2026-02-22T20:16:47.733 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.31610.log.gz 2026-02-22T20:16:47.733 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.25866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34198.log 2026-02-22T20:16:47.733 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37131.log 2026-02-22T20:16:47.733 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19930.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.25866.log.gz 2026-02-22T20:16:47.734 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.34198.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19930.log.gz 2026-02-22T20:16:47.734 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.34198.log.gz 2026-02-22T20:16:47.734 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41412.log 2026-02-22T20:16:47.734 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.37131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32865.log 2026-02-22T20:16:47.735 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.41412.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36711.log 2026-02-22T20:16:47.735 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.37131.log.gz 2026-02-22T20:16:47.735 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.32865.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.25238.log.gz 2026-02-22T20:16:47.735 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.41412.log.gz 2026-02-22T20:16:47.735 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51297.log 2026-02-22T20:16:47.735 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.32865.log.gz 2026-02-22T20:16:47.735 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.36711.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28429.log 2026-02-22T20:16:47.736 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.51297.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36711.log.gz 2026-02-22T20:16:47.736 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25971.log 2026-02-22T20:16:47.736 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42229.log 2026-02-22T20:16:47.736 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.28429.log: /var/log/ceph/cluster1-client.admin.25971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19276.log 2026-02-22T20:16:47.736 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.25971.log.gz 2026-02-22T20:16:47.736 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.28429.log.gz 2026-02-22T20:16:47.736 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.42229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53387.log 2026-02-22T20:16:47.737 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19276.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51297.log.gz 2026-02-22T20:16:47.737 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19276.log.gz 2026-02-22T20:16:47.737 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33983.log 2026-02-22T20:16:47.737 INFO:teuthology.orchestra.run.trial017.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42229.log.gz 2026-02-22T20:16:47.737 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46338.log 2026-02-22T20:16:47.737 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45536.log 2026-02-22T20:16:47.738 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.33983.log: /var/log/ceph/cluster1-client.mirror.46338.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40345.log 2026-02-22T20:16:47.738 INFO:teuthology.orchestra.run.trial017.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53387.log.gz 2026-02-22T20:16:47.738 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46338.log.gz 2026-02-22T20:16:47.738 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.33983.log.gz 2026-02-22T20:16:47.738 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41945.log 2026-02-22T20:16:47.738 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.40345.log: gzip -5 --verbose 95.6% -- -- replaced with /var/log/ceph/cluster1-client.mirror.0.54118.log.gz 2026-02-22T20:16:47.738 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster1-client.mirror.44530.log 2026-02-22T20:16:47.739 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.41945.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41945.log.gz 2026-02-22T20:16:47.739 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.40345.log.gz 2026-02-22T20:16:47.739 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22200.log 2026-02-22T20:16:47.739 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43373.log 2026-02-22T20:16:47.739 INFO:teuthology.orchestra.run.trial017.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44530.log.gz 2026-02-22T20:16:47.740 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.22200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30820.log 2026-02-22T20:16:47.740 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45536.log.gz 2026-02-22T20:16:47.740 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.22200.log.gz 2026-02-22T20:16:47.740 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43015.log 2026-02-22T20:16:47.740 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29113.log 2026-02-22T20:16:47.741 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.30820.log.gz 2026-02-22T20:16:47.741 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54836.log 2026-02-22T20:16:47.741 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43373.log.gz 2026-02-22T20:16:47.741 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.29113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28959.log 2026-02-22T20:16:47.741 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.29113.log.gz 2026-02-22T20:16:47.741 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.54836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53071.log 2026-02-22T20:16:47.742 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54836.log.gz 2026-02-22T20:16:47.742 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.28959.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.28959.log.gz 2026-02-22T20:16:47.742 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53028.log 2026-02-22T20:16:47.742 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.53071.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24446.log 2026-02-22T20:16:47.742 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.53028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36625.log 2026-02-22T20:16:47.743 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53071.log.gz 2026-02-22T20:16:47.743 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24446.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20659.log 2026-02-22T20:16:47.743 INFO:teuthology.orchestra.run.trial017.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.24446.log.gz 2026-02-22T20:16:47.743 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39695.log 2026-02-22T20:16:47.743 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.36625.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53028.log.gz 2026-02-22T20:16:47.743 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.20659.log: 56.7% -- replaced with /var/log/ceph/cluster2-client.admin.20659.log.gz 2026-02-22T20:16:47.743 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.25243.log 2026-02-22T20:16:47.744 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.39695.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21249.log 2026-02-22T20:16:47.744 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39695.log.gz 2026-02-22T20:16:47.744 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.2.25243.log: 19.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.25243.log.gz 2026-02-22T20:16:47.744 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26456.log 2026-02-22T20:16:47.744 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36625.log.gz 2026-02-22T20:16:47.744 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.21249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.54190.log 2026-02-22T20:16:47.745 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.26456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50240.log 2026-02-22T20:16:47.745 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.54190.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.26456.log.gz 2026-02-22T20:16:47.745 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53629.log 2026-02-22T20:16:47.746 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.50240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30504.log 2026-02-22T20:16:47.746 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.53629.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.50240.log.gz 2026-02-22T20:16:47.746 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55250.log 2026-02-22T20:16:47.746 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.53629.log.gz 2026-02-22T20:16:47.746 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29301.log 2026-02-22T20:16:47.747 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.30504.log.gz 2026-02-22T20:16:47.747 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.55250.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59147.log 2026-02-22T20:16:47.747 INFO:teuthology.orchestra.run.trial017.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.admin.55250.log.gz 2026-02-22T20:16:47.747 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.29301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33133.log 2026-02-22T20:16:47.747 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43015.log.gz 2026-02-22T20:16:47.747 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.59147.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.29301.log.gz 2026-02-22T20:16:47.748 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59147.log.gz 2026-02-22T20:16:47.748 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30450.log 2026-02-22T20:16:47.748 INFO:teuthology.orchestra.run.trial017.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.54190.log.gz 2026-02-22T20:16:47.748 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.33133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36177.log 2026-02-22T20:16:47.748 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.30450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52700.log 2026-02-22T20:16:47.748 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.30450.log.gz 2026-02-22T20:16:47.748 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19905.log 2026-02-22T20:16:47.749 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.36177.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33133.log.gz 2026-02-22T20:16:47.749 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.52700.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58910.log 2026-02-22T20:16:47.749 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.36177.log.gz 2026-02-22T20:16:47.749 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.52700.log.gz 2026-02-22T20:16:47.749 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26531.log 2026-02-22T20:16:47.749 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19905.log.gz 2026-02-22T20:16:47.749 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.58910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60029.log 2026-02-22T20:16:47.750 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.26531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48033.log 2026-02-22T20:16:47.750 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.60029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46029.log 2026-02-22T20:16:47.750 INFO:teuthology.orchestra.run.trial017.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.21249.log.gz 87.7% 2026-02-22T20:16:47.750 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.26531.log.gz 2026-02-22T20:16:47.750 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.48033.log: gzip 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60029.log.gz 2026-02-22T20:16:47.751 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54709.log 2026-02-22T20:16:47.751 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48033.log.gz 2026-02-22T20:16:47.751 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.46029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55020.log 2026-02-22T20:16:47.751 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.54709.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46029.log.gz 2026-02-22T20:16:47.751 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46198.log 2026-02-22T20:16:47.752 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.55020.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42203.log 2026-02-22T20:16:47.752 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47150.log 2026-02-22T20:16:47.752 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46198.log: /var/log/ceph/cluster2-client.admin.42203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.54195.log 2026-02-22T20:16:47.752 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.42203.log.gz 2026-02-22T20:16:47.752 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46198.log.gz 2026-02-22T20:16:47.753 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47150.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54709.log.gzgzip 2026-02-22T20:16:47.753 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40455.log 2026-02-22T20:16:47.753 INFO:teuthology.orchestra.run.trial017.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58910.log.gz 2026-02-22T20:16:47.753 INFO:teuthology.orchestra.run.trial017.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55020.log.gz 2026-02-22T20:16:47.753 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47150.log.gz 2026-02-22T20:16:47.753 INFO:teuthology.orchestra.run.trial017.stderr:gzip/var/log/ceph/cluster1-client.mirror.3.54195.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55668.log 2026-02-22T20:16:47.753 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.3.54195.log.gz 2026-02-22T20:16:47.753 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.40455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34351.log 2026-02-22T20:16:47.753 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.55668.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55668.log.gz 2026-02-22T20:16:47.754 INFO:teuthology.orchestra.run.trial017.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.55380.log.gz 2026-02-22T20:16:47.754 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- 87.7% /var/log/ceph/cluster2-client.mirror.50031.log 2026-02-22T20:16:47.754 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40455.log.gz 2026-02-22T20:16:47.754 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.34351.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38308.log 2026-02-22T20:16:47.754 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.34351.log.gz 2026-02-22T20:16:47.754 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39202.log 2026-02-22T20:16:47.754 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.50031.log: /var/log/ceph/cluster2-client.admin.38308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45060.log 2026-02-22T20:16:47.755 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.38308.log.gz 2026-02-22T20:16:47.755 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.39202.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34665.log 2026-02-22T20:16:47.755 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.39202.log.gz 2026-02-22T20:16:47.755 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.45060.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50031.log.gz 2026-02-22T20:16:47.755 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30196.log 2026-02-22T20:16:47.755 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45060.log.gz 2026-02-22T20:16:47.755 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.34665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39074.log 2026-02-22T20:16:47.756 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.34665.log.gz 2026-02-22T20:16:47.756 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37284.log 2026-02-22T20:16:47.756 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.30196.log.gz 2026-02-22T20:16:47.756 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.39074.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35628.log 2026-02-22T20:16:47.757 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.37284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31556.log 2026-02-22T20:16:47.757 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.39074.log.gz 2026-02-22T20:16:47.757 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.35628.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.37284.log.gz 2026-02-22T20:16:47.757 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43400.log 2026-02-22T20:16:47.757 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.31556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42337.log 2026-02-22T20:16:47.757 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.35628.log.gz 2026-02-22T20:16:47.757 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.31556.log.gz 2026-02-22T20:16:47.758 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43400.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31000.log 2026-02-22T20:16:47.758 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.42337.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43400.log.gz 2026-02-22T20:16:47.758 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31583.log 2026-02-22T20:16:47.758 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.42337.log.gz 2026-02-22T20:16:47.758 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.31000.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50216.log 2026-02-22T20:16:47.759 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.31583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.51611.log 2026-02-22T20:16:47.759 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.31000.log.gz 2026-02-22T20:16:47.759 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43903.log 2026-02-22T20:16:47.759 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.50216.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.31583.log.gz 2026-02-22T20:16:47.759 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.51611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51141.log 2026-02-22T20:16:47.759 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50216.log.gz 20.2% 2026-02-22T20:16:47.759 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.51611.log.gz 2026-02-22T20:16:47.759 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.43903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59278.log 2026-02-22T20:16:47.760 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43903.log.gz 2026-02-22T20:16:47.760 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.51141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31086.log 2026-02-22T20:16:47.760 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42445.log 2026-02-22T20:16:47.760 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.59278.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.51141.log.gz 2026-02-22T20:16:47.760 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.31086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52646.log 2026-02-22T20:16:47.761 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.42445.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34144.log 2026-02-22T20:16:47.761 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31086.log.gz 2026-02-22T20:16:47.761 INFO:teuthology.orchestra.run.trial017.stderr: 87.6%/var/log/ceph/cluster1-client.admin.52646.log: -- replaced with /var/log/ceph/cluster1-client.mirror.42445.log.gz 2026-02-22T20:16:47.761 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40087.log 2026-02-22T20:16:47.762 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.52646.log.gz 2026-02-22T20:16:47.762 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43714.log 2026-02-22T20:16:47.762 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.34144.log: /var/log/ceph/cluster1-client.mirror.40087.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45580.log 2026-02-22T20:16:47.762 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.34144.log.gz 2026-02-22T20:16:47.762 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45374.log 2026-02-22T20:16:47.762 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.43714.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40087.log.gz 2026-02-22T20:16:47.762 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36659.log 2026-02-22T20:16:47.763 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45374.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.43714.log.gz 2026-02-22T20:16:47.763 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28302.log 2026-02-22T20:16:47.763 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.36659.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45580.log.gz 2026-02-22T20:16:47.763 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.59380.log 2026-02-22T20:16:47.763 INFO:teuthology.orchestra.run.trial017.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45374.log.gz 2026-02-22T20:16:47.763 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19068.log 2026-02-22T20:16:47.764 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.28302.log: /var/log/ceph/cluster1-client.mirror.2.59380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29995.log 2026-02-22T20:16:47.764 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19068.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19068.log.gz 2026-02-22T20:16:47.764 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34817.log 2026-02-22T20:16:47.764 INFO:teuthology.orchestra.run.trial017.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59278.log.gz 2026-02-22T20:16:47.764 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.28302.log.gz 2026-02-22T20:16:47.764 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36659.log.gz 2026-02-22T20:16:47.764 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.29995.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42778.log 2026-02-22T20:16:47.765 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.29995.log.gz 2026-02-22T20:16:47.765 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.34817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45423.log 2026-02-22T20:16:47.765 INFO:teuthology.orchestra.run.trial017.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.2.59380.log.gz 2026-02-22T20:16:47.765 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47567.log 2026-02-22T20:16:47.765 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.42778.log: /var/log/ceph/cluster2-client.mirror.45423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.48318.log 2026-02-22T20:16:47.766 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47567.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.42778.log.gz 2026-02-22T20:16:47.766 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36735.log 2026-02-22T20:16:47.766 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47567.log.gz 2026-02-22T20:16:47.766 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.34817.log.gz 2026-02-22T20:16:47.766 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.48318.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45423.log.gz 2026-02-22T20:16:47.766 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49049.log 2026-02-22T20:16:47.766 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.36735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53522.log 2026-02-22T20:16:47.767 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.49049.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.36735.log.gz 2026-02-22T20:16:47.767 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23526.log 2026-02-22T20:16:47.767 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53522.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.49049.log.gz 2026-02-22T20:16:47.767 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28147.log 2026-02-22T20:16:47.767 INFO:teuthology.orchestra.run.trial017.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53522.log.gz 2026-02-22T20:16:47.768 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38120.log 2026-02-22T20:16:47.768 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.28147.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.23526.log.gz 2026-02-22T20:16:47.768 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20405.log 2026-02-22T20:16:47.768 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28147.log.gz 2026-02-22T20:16:47.768 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.38120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20431.log 2026-02-22T20:16:47.769 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.38120.log.gz 2026-02-22T20:16:47.769 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.20405.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56307.log 2026-02-22T20:16:47.769 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.20405.log.gz 2026-02-22T20:16:47.769 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.20431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26903.log 2026-02-22T20:16:47.769 INFO:teuthology.orchestra.run.trial017.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.admin.20431.log.gz 2026-02-22T20:16:47.769 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.56307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50470.log 2026-02-22T20:16:47.770 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.26903.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.56307.log.gz 2026-02-22T20:16:47.770 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24932.log 2026-02-22T20:16:47.770 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.26903.log.gz 2026-02-22T20:16:47.770 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.50470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37056.log 2026-02-22T20:16:47.770 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24932.log: 83.7% -- replaced with /var/log/ceph/cluster2-client.mirror.24932.log.gz 2026-02-22T20:16:47.771 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41016.log 2026-02-22T20:16:47.771 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.37056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29750.log 2026-02-22T20:16:47.771 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50470.log.gz 2026-02-22T20:16:47.771 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37056.log.gz 2026-02-22T20:16:47.771 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.41016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22415.log 2026-02-22T20:16:47.772 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.41016.log.gz 2026-02-22T20:16:47.772 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.29750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30423.log 2026-02-22T20:16:47.772 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.22415.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.29750.log.gz 2026-02-22T20:16:47.772 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44068.log 2026-02-22T20:16:47.772 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31919.log 2026-02-22T20:16:47.773 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.30423.log: /var/log/ceph/cluster2-client.mirror.44068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37550.log 2026-02-22T20:16:47.773 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.30423.log.gz 2026-02-22T20:16:47.773 INFO:teuthology.orchestra.run.trial017.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44068.log.gz 2026-02-22T20:16:47.773 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.31919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33803.log 2026-02-22T20:16:47.773 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.31919.log.gz 2026-02-22T20:16:47.773 INFO:teuthology.orchestra.run.trial017.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.22415.log.gz 2026-02-22T20:16:47.774 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.37550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43146.log 2026-02-22T20:16:47.774 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.33803.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.37550.log.gz 2026-02-22T20:16:47.774 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.33803.log.gz 2026-02-22T20:16:47.774 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34746.log 2026-02-22T20:16:47.774 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22914.log 2026-02-22T20:16:47.775 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.34746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22532.log 2026-02-22T20:16:47.775 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.34746.log.gz/var/log/ceph/cluster1-client.admin.22914.log: 2026-02-22T20:16:47.775 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43146.log.gz 2026-02-22T20:16:47.775 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44444.log 2026-02-22T20:16:47.775 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.22914.log.gz 2026-02-22T20:16:47.775 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.22532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38832.log 2026-02-22T20:16:47.776 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.22532.log.gz 2026-02-22T20:16:47.776 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35921.log 2026-02-22T20:16:47.776 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44444.log.gz 2026-02-22T20:16:47.776 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.38832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32978.log 2026-02-22T20:16:47.776 INFO:teuthology.orchestra.run.trial017.stderr: 87.7%/var/log/ceph/cluster1-client.mirror.35921.log: -- replaced with /var/log/ceph/cluster2-client.admin.38832.log.gz 2026-02-22T20:16:47.777 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49404.log 2026-02-22T20:16:47.777 INFO:teuthology.orchestra.run.trial017.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.48318.log.gz 2026-02-22T20:16:47.777 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.32978.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38969.log 2026-02-22T20:16:47.777 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.32978.log.gz 2026-02-22T20:16:47.777 INFO:teuthology.orchestra.run.trial017.stderr:gzip/var/log/ceph/cluster1-client.mirror.49404.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.37577.log 2026-02-22T20:16:47.778 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.38969.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49404.log.gz 2026-02-22T20:16:47.778 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.35921.log.gz 2026-02-22T20:16:47.778 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38969.log.gz 2026-02-22T20:16:47.778 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33752.log 2026-02-22T20:16:47.778 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.37577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26585.log 2026-02-22T20:16:47.778 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.33752.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.37577.log.gz 2026-02-22T20:16:47.778 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26951.log 2026-02-22T20:16:47.779 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33752.log.gz 2026-02-22T20:16:47.779 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.26585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23247.log 2026-02-22T20:16:47.779 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.26951.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.26585.log.gz 2026-02-22T20:16:47.779 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45102.log 2026-02-22T20:16:47.780 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23247.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.26951.log.gz 2026-02-22T20:16:47.780 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29086.log 2026-02-22T20:16:47.780 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24303.log 2026-02-22T20:16:47.780 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.23247.log.gz 2026-02-22T20:16:47.780 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.29086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47259.log 2026-02-22T20:16:47.781 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.24303.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.29086.log.gz 2026-02-22T20:16:47.781 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2026-02-22T20:16:47.781 INFO:teuthology.orchestra.run.trial017.stderr: 87.6%/var/log/ceph/cluster2-client.mirror.47259.log: -- replaced with /var/log/ceph/cluster1-client.admin.24303.log.gz 2026-02-22T20:16:47.781 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38945.log 2026-02-22T20:16:47.781 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47193.log 2026-02-22T20:16:47.782 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47259.log.gz 2026-02-22T20:16:47.782 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.38945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55851.log 2026-02-22T20:16:47.782 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38945.log.gz 2026-02-22T20:16:47.782 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51567.log 2026-02-22T20:16:47.783 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.55851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39719.log 2026-02-22T20:16:47.783 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55851.log.gz 2026-02-22T20:16:47.783 INFO:teuthology.orchestra.run.trial017.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47193.log.gz 2026-02-22T20:16:47.783 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.51567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28857.log 2026-02-22T20:16:47.783 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.39719.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32548.log 2026-02-22T20:16:47.784 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39719.log.gz 2026-02-22T20:16:47.784 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51567.log.gz 2026-02-22T20:16:47.784 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.28857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21400.log 2026-02-22T20:16:47.784 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.32548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50911.log 2026-02-22T20:16:47.784 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.21400.log: 87.2% -- replaced with /var/log/ceph/cluster2.log.gz 2026-02-22T20:16:47.785 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.28857.log.gz 2026-02-22T20:16:47.785 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43100.log 2026-02-22T20:16:47.785 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.32548.log.gz 2026-02-22T20:16:47.785 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.21400.log.gz 2026-02-22T20:16:47.785 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.50911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41753.log 2026-02-22T20:16:47.785 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43100.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.50911.log.gz 2026-02-22T20:16:47.785 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42886.log 2026-02-22T20:16:47.786 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.41753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52780.log 2026-02-22T20:16:47.786 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45102.log.gz 2026-02-22T20:16:47.786 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.41753.log.gz 2026-02-22T20:16:47.786 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.42886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54729.log 2026-02-22T20:16:47.786 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.42886.log.gz 2026-02-22T20:16:47.786 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23716.log 2026-02-22T20:16:47.787 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.54729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31053.log 2026-02-22T20:16:47.787 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43100.log.gz 2026-02-22T20:16:47.787 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44873.log 2026-02-22T20:16:47.788 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.31053.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22308.log 2026-02-22T20:16:47.788 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.23716.log.gz 2026-02-22T20:16:47.788 INFO:teuthology.orchestra.run.trial017.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52780.log.gz 2026-02-22T20:16:47.788 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44873.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45875.log 2026-02-22T20:16:47.788 INFO:teuthology.orchestra.run.trial017.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54729.log.gz 2026-02-22T20:16:47.788 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44873.log.gz 2026-02-22T20:16:47.788 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31053.log.gz 2026-02-22T20:16:47.788 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60553.log 2026-02-22T20:16:47.789 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.22308.log: /var/log/ceph/cluster2-client.mirror.45875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23796.log 2026-02-22T20:16:47.789 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.22308.log.gz 2026-02-22T20:16:47.789 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16379.log 2026-02-22T20:16:47.789 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60553.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60553.log.gz 2026-02-22T20:16:47.789 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31636.log 2026-02-22T20:16:47.789 INFO:teuthology.orchestra.run.trial017.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.23796.log.gz 2026-02-22T20:16:47.790 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.16379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29909.log 2026-02-22T20:16:47.790 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16379.log.gz 2026-02-22T20:16:47.790 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38517.log 2026-02-22T20:16:47.790 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.31636.log: /var/log/ceph/cluster2-client.admin.29909.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43424.log 2026-02-22T20:16:47.790 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.29909.log.gz 2026-02-22T20:16:47.791 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.38517.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31636.log.gz 2026-02-22T20:16:47.791 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35099.log 2026-02-22T20:16:47.791 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.43424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30578.log 2026-02-22T20:16:47.791 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43424.log.gz 2026-02-22T20:16:47.791 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.35099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.54109.log 2026-02-22T20:16:47.791 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38517.log.gz 2026-02-22T20:16:47.792 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.35099.log.gz 2026-02-22T20:16:47.792 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30578.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28932.log 2026-02-22T20:16:47.792 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.54109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44152.log 2026-02-22T20:16:47.792 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.30578.log.gz 2026-02-22T20:16:47.792 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.54109.log.gz 2026-02-22T20:16:47.792 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.28932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58539.log 2026-02-22T20:16:47.793 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.28932.log.gz 2026-02-22T20:16:47.793 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60683.log 2026-02-22T20:16:47.793 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40632.log 2026-02-22T20:16:47.793 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.58539.log: /var/log/ceph/cluster2-client.admin.60683.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58539.log.gz 2026-02-22T20:16:47.793 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60683.log.gz 2026-02-22T20:16:47.793 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53186.log 2026-02-22T20:16:47.794 INFO:teuthology.orchestra.run.trial017.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44152.log.gz 2026-02-22T20:16:47.794 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.40632.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40632.log.gz 2026-02-22T20:16:47.794 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60605.log 2026-02-22T20:16:47.794 INFO:teuthology.orchestra.run.trial017.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45875.log.gz 2026-02-22T20:16:47.794 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.53186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46504.log 2026-02-22T20:16:47.794 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45664.log 2026-02-22T20:16:47.795 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60605.log.gz 2026-02-22T20:16:47.795 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19425.log 2026-02-22T20:16:47.795 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46504.log.gz 2026-02-22T20:16:47.795 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35604.log 2026-02-22T20:16:47.796 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.admin.19425.log: /var/log/ceph/cluster2-client.mirror.35973.log 2026-02-22T20:16:47.796 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19425.log.gz 2026-02-22T20:16:47.796 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.35604.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45664.log.gz 2026-02-22T20:16:47.796 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.35604.log.gz 2026-02-22T20:16:47.796 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32712.log 2026-02-22T20:16:47.796 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37364.log 2026-02-22T20:16:47.797 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.35973.log: /var/log/ceph/cluster1-client.admin.32712.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35973.log.gz 2026-02-22T20:16:47.797 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.32712.log.gz 2026-02-22T20:16:47.797 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47281.log 2026-02-22T20:16:47.797 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46735.log 2026-02-22T20:16:47.797 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.37364.log: /var/log/ceph/cluster2-client.mirror.47281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.20740.log 2026-02-22T20:16:47.798 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.21799.log 2026-02-22T20:16:47.798 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37364.log.gz 2026-02-22T20:16:47.798 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.20740.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46735.log.gz 2026-02-22T20:16:47.798 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47281.log.gz 2026-02-22T20:16:47.798 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45218.log 2026-02-22T20:16:47.798 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.21799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23193.log 2026-02-22T20:16:47.798 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.21799.log.gz 2026-02-22T20:16:47.799 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.45218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49177.log 2026-02-22T20:16:47.799 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45218.log.gz 2026-02-22T20:16:47.799 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29355.log 2026-02-22T20:16:47.799 INFO:teuthology.orchestra.run.trial017.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53186.log.gz 2026-02-22T20:16:47.799 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.23193.log.gz 2026-02-22T20:16:47.799 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.49177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23094.log 2026-02-22T20:16:47.800 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54994.log 2026-02-22T20:16:47.800 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.29355.log: /var/log/ceph/cluster1-client.admin.23094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32000.log 2026-02-22T20:16:47.800 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.29355.log.gz 2026-02-22T20:16:47.800 INFO:teuthology.orchestra.run.trial017.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.23094.log.gz 2026-02-22T20:16:47.800 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.54146.log 2026-02-22T20:16:47.801 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.54994.log: /var/log/ceph/cluster1-client.admin.32000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44241.log 2026-02-22T20:16:47.801 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.32000.log.gz 2026-02-22T20:16:47.801 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.54994.log.gz 2026-02-22T20:16:47.801 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.54146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30874.log 2026-02-22T20:16:47.801 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.49177.log.gz 2026-02-22T20:16:47.801 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44241.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.20740.log.gz 2026-02-22T20:16:47.801 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47435.log 2026-02-22T20:16:47.802 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.30874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52340.log 2026-02-22T20:16:47.802 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30793.log 2026-02-22T20:16:47.802 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.30874.log.gz 2026-02-22T20:16:47.802 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44241.log.gz 2026-02-22T20:16:47.802 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43642.log 2026-02-22T20:16:47.803 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52340.log.gz 2026-02-22T20:16:47.803 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47435.log.gz 2026-02-22T20:16:47.803 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19705.log 2026-02-22T20:16:47.803 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43642.log: gzip 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.30793.log.gz -5 2026-02-22T20:16:47.803 INFO:teuthology.orchestra.run.trial017.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.43273.log 2026-02-22T20:16:47.803 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19705.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19705.log.gz 2026-02-22T20:16:47.804 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47105.log 2026-02-22T20:16:47.804 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43642.log.gz 93.4% 2026-02-22T20:16:47.804 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.1.54146.log.gz 2026-02-22T20:16:47.804 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34378.log 2026-02-22T20:16:47.804 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.47105.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47105.log.gz 2026-02-22T20:16:47.804 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45621.log 2026-02-22T20:16:47.805 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37311.log 2026-02-22T20:16:47.805 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.34378.log: /var/log/ceph/cluster2-client.mirror.45621.log: gzip 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.34378.log.gz 2026-02-22T20:16:47.805 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45394.log 2026-02-22T20:16:47.805 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.37311.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23553.log 2026-02-22T20:16:47.806 INFO:teuthology.orchestra.run.trial017.stderr: 87.6%/var/log/ceph/cluster1-client.mirror.45394.log: -- replaced with /var/log/ceph/cluster2-client.admin.37311.log.gz 2026-02-22T20:16:47.806 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40916.log 2026-02-22T20:16:47.806 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42805.log 2026-02-22T20:16:47.807 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.40916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46240.log 2026-02-22T20:16:47.807 INFO:teuthology.orchestra.run.trial017.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.23553.log.gz 2026-02-22T20:16:47.807 INFO:teuthology.orchestra.run.trial017.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45394.log.gz 2026-02-22T20:16:47.807 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.42805.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40916.log.gz 2026-02-22T20:16:47.807 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53747.log 2026-02-22T20:16:47.807 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49865.log 2026-02-22T20:16:47.808 INFO:teuthology.orchestra.run.trial017.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.admin.42805.log.gz 2026-02-22T20:16:47.808 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44725.log 2026-02-22T20:16:47.808 INFO:teuthology.orchestra.run.trial017.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53747.log.gz 2026-02-22T20:16:47.808 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.49865.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16403.log 2026-02-22T20:16:47.808 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.49865.log.gz 2026-02-22T20:16:47.808 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22707.log 2026-02-22T20:16:47.809 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.16403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49329.log 2026-02-22T20:16:47.809 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16403.log.gz 2026-02-22T20:16:47.809 INFO:teuthology.orchestra.run.trial017.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44725.log.gz 2026-02-22T20:16:47.809 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.22707.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.22707.log.gz 2026-02-22T20:16:47.809 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60501.log 2026-02-22T20:16:47.810 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.49329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22788.log 2026-02-22T20:16:47.810 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22887.log 2026-02-22T20:16:47.810 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60501.log.gz 2026-02-22T20:16:47.810 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49329.log.gz 2026-02-22T20:16:47.810 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.22788.log: 94.0%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.45621.log.gz 2026-02-22T20:16:47.810 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43817.log 2026-02-22T20:16:47.811 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.22887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31268.log 2026-02-22T20:16:47.811 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.22788.log.gz 2026-02-22T20:16:47.811 INFO:teuthology.orchestra.run.trial017.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43273.log.gz 2026-02-22T20:16:47.811 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27405.log 2026-02-22T20:16:47.811 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.22887.log.gz 2026-02-22T20:16:47.811 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.31268.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44402.log 2026-02-22T20:16:47.812 INFO:teuthology.orchestra.run.trial017.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43817.log.gz 2026-02-22T20:16:47.812 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30124.log 2026-02-22T20:16:47.812 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.27405.log: /var/log/ceph/cluster2-client.mirror.44402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35997.log 2026-02-22T20:16:47.812 INFO:teuthology.orchestra.run.trial017.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46240.log.gz 2026-02-22T20:16:47.812 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.27405.log.gz 2026-02-22T20:16:47.812 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31268.log.gz 2026-02-22T20:16:47.813 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.30124.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50012.log 2026-02-22T20:16:47.813 INFO:teuthology.orchestra.run.trial017.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44402.log.gz 2026-02-22T20:16:47.813 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.30124.log.gz 2026-02-22T20:16:47.813 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.35997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60449.log 2026-02-22T20:16:47.813 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.35997.log.gz 2026-02-22T20:16:47.813 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.50012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28717.log 2026-02-22T20:16:47.813 INFO:teuthology.orchestra.run.trial017.stderr: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50012.log.gz 2026-02-22T20:16:47.813 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60449.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60449.log.gz 2026-02-22T20:16:47.813 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52622.log 2026-02-22T20:16:47.814 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.28717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.59888.log 2026-02-22T20:16:47.814 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.28717.log.gz 2026-02-22T20:16:47.814 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37211.log 2026-02-22T20:16:47.814 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.59888.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52622.log.gz 2026-02-22T20:16:47.814 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20102.log 2026-02-22T20:16:47.814 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.59888.log.gz 2026-02-22T20:16:47.815 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.37211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55272.log 2026-02-22T20:16:47.815 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.20102.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40862.log 2026-02-22T20:16:47.815 INFO:teuthology.orchestra.run.trial017.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.20102.log.gz 2026-02-22T20:16:47.815 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.55272.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31844.log 2026-02-22T20:16:47.815 INFO:teuthology.orchestra.run.trial017.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37211.log.gz 2026-02-22T20:16:47.816 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.40862.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.55272.log.gz 2026-02-22T20:16:47.816 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20518.log 2026-02-22T20:16:47.816 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.40862.log.gz 2026-02-22T20:16:47.816 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.31844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38732.log 2026-02-22T20:16:47.816 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.20518.log: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.20518.log.gz 2026-02-22T20:16:47.816 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28670.log 2026-02-22T20:16:47.817 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.38732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34868.log 2026-02-22T20:16:47.817 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.28670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54815.log 2026-02-22T20:16:47.818 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31844.log.gz 2026-02-22T20:16:47.818 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38732.log.gz 2026-02-22T20:16:47.818 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.28670.log.gz 2026-02-22T20:16:47.818 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.34868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32315.log 2026-02-22T20:16:47.818 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.34868.log.gz 2026-02-22T20:16:47.818 INFO:teuthology.orchestra.run.trial017.stderr:gzip/var/log/ceph/cluster2-client.mirror.54815.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.30250.log 2026-02-22T20:16:47.818 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.32315.log: gzip 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.32315.log.gz -5 2026-02-22T20:16:47.818 INFO:teuthology.orchestra.run.trial017.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.3.57418.log 2026-02-22T20:16:47.819 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.30250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35126.log 2026-02-22T20:16:47.819 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54815.log.gz 2026-02-22T20:16:47.819 INFO:teuthology.orchestra.run.trial017.stderr: 87.6%/var/log/ceph/cluster1-client.mirror.3.57418.log: -- replaced with /var/log/ceph/cluster2-client.admin.30250.log.gz 2026-02-22T20:16:47.819 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42625.log 2026-02-22T20:16:47.819 INFO:teuthology.orchestra.run.trial017.stderr: 19.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.57418.log.gz 2026-02-22T20:16:47.819 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.35126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47781.log 2026-02-22T20:16:47.820 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 87.8% --verbose -- -- replaced with /var/log/ceph/cluster2-client.admin.35126.log.gz 2026-02-22T20:16:47.820 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster1-client.mirror.35152.log 2026-02-22T20:16:47.820 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.42625.log: /var/log/ceph/cluster2-client.mirror.47781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58830.log 2026-02-22T20:16:47.820 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47781.log.gz 2026-02-22T20:16:47.820 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.35152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19680.log 2026-02-22T20:16:47.821 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42625.log.gz 2026-02-22T20:16:47.821 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42122.log 2026-02-22T20:16:47.821 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.58830.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.35152.log.gz 2026-02-22T20:16:47.821 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19680.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19680.log.gz 2026-02-22T20:16:47.821 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48287.log 2026-02-22T20:16:47.821 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.58830.log.gz 2026-02-22T20:16:47.821 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28908.log 2026-02-22T20:16:47.822 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.42122.log: /var/log/ceph/cluster2-client.mirror.48287.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42122.log.gz 2026-02-22T20:16:47.822 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42364.log 2026-02-22T20:16:47.822 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.28908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44466.log 2026-02-22T20:16:47.822 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48287.log.gz 2026-02-22T20:16:47.823 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.42364.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.28908.log.gz 2026-02-22T20:16:47.823 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45898.log 2026-02-22T20:16:47.823 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44466.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.42364.log.gz 2026-02-22T20:16:47.823 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37993.log 2026-02-22T20:16:47.823 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44466.log.gz 2026-02-22T20:16:47.823 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49811.log 2026-02-22T20:16:47.824 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.37993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33929.log 2026-02-22T20:16:47.824 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45898.log.gz 2026-02-22T20:16:47.824 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.49811.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37993.log.gz 2026-02-22T20:16:47.824 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51244.log 2026-02-22T20:16:47.824 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.33929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47866.log 2026-02-22T20:16:47.824 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.49811.log.gz 2026-02-22T20:16:47.825 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.33929.log.gz 2026-02-22T20:16:47.825 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.51244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24093.log 2026-02-22T20:16:47.825 INFO:teuthology.orchestra.run.trial017.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.admin.51244.log.gz 2026-02-22T20:16:47.825 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54796.log 2026-02-22T20:16:47.825 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47866.log: /var/log/ceph/cluster1-client.admin.24093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47037.log 2026-02-22T20:16:47.825 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47866.log.gz 2026-02-22T20:16:47.826 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.54796.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.24093.log.gz 2026-02-22T20:16:47.826 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46070.log 2026-02-22T20:16:47.826 INFO:teuthology.orchestra.run.trial017.stderr: 83.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54796.log.gz 2026-02-22T20:16:47.826 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.47037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28276.log 2026-02-22T20:16:47.826 INFO:teuthology.orchestra.run.trial017.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47037.log.gz 2026-02-22T20:16:47.827 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46070.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.51648.log 2026-02-22T20:16:47.827 INFO:teuthology.orchestra.run.trial017.stderr: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46070.log.gz 2026-02-22T20:16:47.827 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.28276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30847.log 2026-02-22T20:16:47.827 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.28276.log.gz 2026-02-22T20:16:47.827 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.51648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43496.log 2026-02-22T20:16:47.828 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.30847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45984.log 2026-02-22T20:16:47.828 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.43496.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16527.log 2026-02-22T20:16:47.828 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.30847.log.gz 2026-02-22T20:16:47.828 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.45984.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.43496.log.gz 2026-02-22T20:16:47.828 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39255.log 2026-02-22T20:16:47.829 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45984.log.gz 2026-02-22T20:16:47.829 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.16527.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49892.log 2026-02-22T20:16:47.829 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16527.log.gz 2026-02-22T20:16:47.829 INFO:teuthology.orchestra.run.trial017.stderr:gzip/var/log/ceph/cluster2-client.mirror.39255.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.37649.log 2026-02-22T20:16:47.830 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.49892.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50294.log 2026-02-22T20:16:47.830 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.37649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23769.log 2026-02-22T20:16:47.830 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.admin.49892.log.gz 2026-02-22T20:16:47.830 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39255.log.gz 2026-02-22T20:16:47.830 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.37649.log.gz 2026-02-22T20:16:47.830 INFO:teuthology.orchestra.run.trial017.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.51648.log.gz 2026-02-22T20:16:47.830 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.50294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.50576.log 2026-02-22T20:16:47.831 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.50294.log.gz 2026-02-22T20:16:47.831 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24911.log 2026-02-22T20:16:47.831 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.23769.log.gz 2026-02-22T20:16:47.831 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.50576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27432.log 2026-02-22T20:16:47.831 INFO:teuthology.orchestra.run.trial017.stderr: 20.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.50576.log.gz 2026-02-22T20:16:47.831 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24911.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24911.log.gz 2026-02-22T20:16:47.832 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52126.log 2026-02-22T20:16:47.832 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.27432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52829.log 2026-02-22T20:16:47.832 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52126.log: 87.4%gzip -- replaced with /var/log/ceph/cluster1-client.admin.27432.log.gz 2026-02-22T20:16:47.832 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.38652.log 2026-02-22T20:16:47.833 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52829.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52126.log.gz 2026-02-22T20:16:47.833 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28348.log 2026-02-22T20:16:47.833 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.38652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25944.log 2026-02-22T20:16:47.833 INFO:teuthology.orchestra.run.trial017.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52829.log.gz 2026-02-22T20:16:47.833 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.38652.log.gz 2026-02-22T20:16:47.833 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.28348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42176.log 2026-02-22T20:16:47.834 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.25944.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.28348.log.gz 2026-02-22T20:16:47.834 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.21549.log 2026-02-22T20:16:47.834 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.25944.log.gz 2026-02-22T20:16:47.834 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.48346.log 2026-02-22T20:16:47.834 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.42176.log: /var/log/ceph/cluster1-client.mirror.21549.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29535.log 2026-02-22T20:16:47.834 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.42176.log.gz 2026-02-22T20:16:47.835 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.21549.log.gz 2026-02-22T20:16:47.835 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.48346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25638.log 2026-02-22T20:16:47.835 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.29535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33857.log 2026-02-22T20:16:47.835 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.29535.log.gz 2026-02-22T20:16:47.836 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.25638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57994.log 2026-02-22T20:16:47.836 INFO:teuthology.orchestra.run.trial017.stderr: 87.8%/var/log/ceph/cluster2-client.admin.33857.log: -- replaced with /var/log/ceph/cluster1-client.admin.25638.log.gz 2026-02-22T20:16:47.836 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55534.log 2026-02-22T20:16:47.836 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57994.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.33857.log.gz 2026-02-22T20:16:47.836 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose 87.7% -- -- replaced with /var/log/ceph/cluster1-client.mirror.57994.log.gz 2026-02-22T20:16:47.836 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster1-client.mirror.26253.log 2026-02-22T20:16:47.837 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.55534.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2026-02-22T20:16:47.837 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55534.log.gz 2026-02-22T20:16:47.837 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.26253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40533.log 2026-02-22T20:16:47.837 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose/var/log/ceph/cluster2.audit.log: -- /var/log/ceph/cluster1-client.admin.58803.log 2026-02-22T20:16:47.838 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.40533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32395.log 2026-02-22T20:16:47.838 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.40533.log.gz 2026-02-22T20:16:47.838 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.58803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55901.log 2026-02-22T20:16:47.838 INFO:teuthology.orchestra.run.trial017.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.26253.log.gz 2026-02-22T20:16:47.838 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.32395.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.58803.log.gz 2026-02-22T20:16:47.839 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39482.log 2026-02-22T20:16:47.839 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.55901.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47325.log 2026-02-22T20:16:47.839 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55901.log.gz 2026-02-22T20:16:47.839 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.39482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24420.log 2026-02-22T20:16:47.839 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32395.log.gz 2026-02-22T20:16:47.840 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.20828.log 2026-02-22T20:16:47.840 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47325.log.gz 2026-02-22T20:16:47.840 INFO:teuthology.orchestra.run.trial017.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.48346.log.gz 2026-02-22T20:16:47.840 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.24420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24509.log 2026-02-22T20:16:47.840 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39482.log.gz 2026-02-22T20:16:47.840 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.20828.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.24420.log.gz 2026-02-22T20:16:47.840 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38625.log 2026-02-22T20:16:47.841 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32811.log 2026-02-22T20:16:47.841 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.24509.log.gz 2026-02-22T20:16:47.841 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.38625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49621.log 2026-02-22T20:16:47.841 INFO:teuthology.orchestra.run.trial017.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.20828.log.gz 2026-02-22T20:16:47.842 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.32811.log: gzip -5 88.5% --verbose -- replaced with /var/log/ceph/cluster2-client.admin.38625.log.gz -- 2026-02-22T20:16:47.842 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster1-client.mirror.29645.log 2026-02-22T20:16:47.842 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.48351.log 2026-02-22T20:16:47.842 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.32811.log.gz 2026-02-22T20:16:47.842 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.49621.log: /var/log/ceph/cluster1-client.mirror.29645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42467.log 2026-02-22T20:16:47.842 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.49621.log.gz 2026-02-22T20:16:47.842 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.29645.log.gz 2026-02-22T20:16:47.843 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.48351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46316.log 2026-02-22T20:16:47.843 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.48351.log.gz 2026-02-22T20:16:47.843 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.42467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35314.log 2026-02-22T20:16:47.843 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.46316.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42467.log.gz 2026-02-22T20:16:47.843 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32288.log 2026-02-22T20:16:47.843 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46316.log.gz 2026-02-22T20:16:47.844 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.35314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53478.log 2026-02-22T20:16:47.844 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.32288.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.35314.log.gz 2026-02-22T20:16:47.844 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37778.log 2026-02-22T20:16:47.844 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.32288.log.gz 2026-02-22T20:16:47.844 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53478.log: gzip -5 94.0% --verbose -- replaced with /var/log/ceph/cluster2.audit.log.gz -- 2026-02-22T20:16:47.844 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster2-client.admin.16462.log 2026-02-22T20:16:47.845 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.37778.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53478.log.gz 2026-02-22T20:16:47.845 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30745.log 2026-02-22T20:16:47.845 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37778.log.gz 2026-02-22T20:16:47.845 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.16462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47845.log 0.0% 2026-02-22T20:16:47.845 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.16462.log.gz 2026-02-22T20:16:47.846 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.30745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53008.log 2026-02-22T20:16:47.846 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.30745.log.gz 2026-02-22T20:16:47.846 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40889.log 2026-02-22T20:16:47.846 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.53008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53568.log 2026-02-22T20:16:47.847 INFO:teuthology.orchestra.run.trial017.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53008.log.gz 2026-02-22T20:16:47.847 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.40889.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47845.log.gz 2026-02-22T20:16:47.847 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22583.log 2026-02-22T20:16:47.847 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.40889.log.gz 2026-02-22T20:16:47.847 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53568.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52760.log 2026-02-22T20:16:47.847 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53568.log.gz 2026-02-22T20:16:47.848 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.22583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37862.log 2026-02-22T20:16:47.848 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28249.log 2026-02-22T20:16:47.848 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.22583.log.gz 2026-02-22T20:16:47.848 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25665.log 2026-02-22T20:16:47.848 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.37862.log: /var/log/ceph/cluster1-client.admin.28249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26134.log 2026-02-22T20:16:47.849 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37862.log.gz 2026-02-22T20:16:47.849 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.25665.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.28249.log.gz 2026-02-22T20:16:47.849 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.25665.log.gz 2026-02-22T20:16:47.849 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44220.log 2026-02-22T20:16:47.849 INFO:teuthology.orchestra.run.trial017.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52760.log.gz 2026-02-22T20:16:47.849 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.26134.log: -- /var/log/ceph/cluster1-client.mirror.20136.log 2026-02-22T20:16:47.850 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44220.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.26134.log.gz 2026-02-22T20:16:47.850 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27611.log 2026-02-22T20:16:47.850 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.20136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51077.log 2026-02-22T20:16:47.850 INFO:teuthology.orchestra.run.trial017.stderr: 54.4% -- replaced with /var/log/ceph/cluster1-client.mirror.20136.log.gz 2026-02-22T20:16:47.850 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44220.log.gz 2026-02-22T20:16:47.850 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.27611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28222.log 2026-02-22T20:16:47.851 INFO:teuthology.orchestra.run.trial017.stderr: 87.6%/var/log/ceph/cluster1-client.admin.51077.log: -- replaced with /var/log/ceph/cluster1-client.admin.27611.log.gz 2026-02-22T20:16:47.851 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36419.log 2026-02-22T20:16:47.851 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.28222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27181.log 2026-02-22T20:16:47.851 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.51077.log.gz 2026-02-22T20:16:47.851 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.28222.log.gz 2026-02-22T20:16:47.851 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.36419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53682.log 2026-02-22T20:16:47.852 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.27181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30399.log 2026-02-22T20:16:47.852 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.36419.log.gz 2026-02-22T20:16:47.852 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53682.log: 87.9%gzip -5 -- replaced with /var/log/ceph/cluster1-client.admin.27181.log.gz --verbose 2026-02-22T20:16:47.852 INFO:teuthology.orchestra.run.trial017.stderr: -- /var/log/ceph/cluster1-client.mirror.54510.log 2026-02-22T20:16:47.852 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32100.log 2026-02-22T20:16:47.853 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.30399.log: /var/log/ceph/cluster1-client.mirror.54510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24761.log 2026-02-22T20:16:47.853 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.30399.log.gz 2026-02-22T20:16:47.853 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54510.log.gz 2026-02-22T20:16:47.853 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35888.log 2026-02-22T20:16:47.853 INFO:teuthology.orchestra.run.trial017.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53682.log.gz 2026-02-22T20:16:47.853 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.32100.log: /var/log/ceph/cluster1-client.mirror.24761.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.24761.log.gz 2026-02-22T20:16:47.853 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24393.log 2026-02-22T20:16:47.853 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.32100.log.gz 2026-02-22T20:16:47.854 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33488.log 2026-02-22T20:16:47.854 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.35888.log: /var/log/ceph/cluster1-client.admin.24393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25044.log 2026-02-22T20:16:47.854 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.33488.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.24393.log.gz 2026-02-22T20:16:47.854 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41492.log 2026-02-22T20:16:47.854 INFO:teuthology.orchestra.run.trial017.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.33488.log.gz/var/log/ceph/cluster2-client.mirror.25044.log: 2026-02-22T20:16:47.854 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24711.log 2026-02-22T20:16:47.854 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.35888.log.gz 2026-02-22T20:16:47.855 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54649.log 2026-02-22T20:16:47.855 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.41492.log: /var/log/ceph/cluster2-client.mirror.24711.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40138.log 2026-02-22T20:16:47.855 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.25044.log.gz 2026-02-22T20:16:47.855 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.54649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40969.log 2026-02-22T20:16:47.856 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.54649.log.gz 2026-02-22T20:16:47.856 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41492.log.gz 2026-02-22T20:16:47.856 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.40138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23994.log 2026-02-22T20:16:47.856 INFO:teuthology.orchestra.run.trial017.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.24711.log.gz 2026-02-22T20:16:47.856 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.40138.log.gz 2026-02-22T20:16:47.856 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.40969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39668.log 2026-02-22T20:16:47.857 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38281.log 2026-02-22T20:16:47.857 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.39668.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.23994.log.gz 2026-02-22T20:16:47.857 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39770.log 2026-02-22T20:16:47.857 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.38281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.57385.log 2026-02-22T20:16:47.857 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40969.log.gz 2026-02-22T20:16:47.857 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.38281.log.gz 2026-02-22T20:16:47.857 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.39770.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.39668.log.gz 2026-02-22T20:16:47.857 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.59457.log 2026-02-22T20:16:47.858 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.39770.log.gz 2026-02-22T20:16:47.858 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31113.log 2026-02-22T20:16:47.858 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.57385.log: /var/log/ceph/cluster1-client.mirror.3.59457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44174.log 2026-02-22T20:16:47.858 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.31113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52301.log 2026-02-22T20:16:47.858 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31113.log.gz 2026-02-22T20:16:47.859 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38146.log 2026-02-22T20:16:47.859 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27934.log 2026-02-22T20:16:47.859 INFO:teuthology.orchestra.run.trial017.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44174.log.gz 2026-02-22T20:16:47.859 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52301.log.gz 2026-02-22T20:16:47.859 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41123.log 2026-02-22T20:16:47.860 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.38146.log: /var/log/ceph/cluster1-client.mirror.27934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54906.log 2026-02-22T20:16:47.860 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.41123.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.27934.log.gz 2026-02-22T20:16:47.860 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27881.log 2026-02-22T20:16:47.860 INFO:teuthology.orchestra.run.trial017.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.59457.log.gz 2026-02-22T20:16:47.860 INFO:teuthology.orchestra.run.trial017.stderr: 89.3%/var/log/ceph/cluster1-client.admin.54906.log: -- replaced with /var/log/ceph/cluster1-client.mirror.38146.log.gz 2026-02-22T20:16:47.861 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.54906.log.gz 2026-02-22T20:16:47.861 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59861.log 2026-02-22T20:16:47.861 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41123.log.gz 2026-02-22T20:16:47.861 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.27881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24886.log 2026-02-22T20:16:47.861 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.59861.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.27881.log.gz 2026-02-22T20:16:47.861 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30043.log 2026-02-22T20:16:47.862 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.59861.log.gz 2026-02-22T20:16:47.862 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24886.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24951.log 2026-02-22T20:16:47.862 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24886.log.gz 2026-02-22T20:16:47.862 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28617.log 2026-02-22T20:16:47.862 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30043.log: /var/log/ceph/cluster2-client.mirror.24951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25072.log 2026-02-22T20:16:47.862 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.30043.log.gz 2026-02-22T20:16:47.862 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.24951.log.gz 2026-02-22T20:16:47.863 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.28617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35808.log 2026-02-22T20:16:47.863 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.25072.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.28617.log.gz 2026-02-22T20:16:47.863 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.25072.log.gz 2026-02-22T20:16:47.863 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37472.log 2026-02-22T20:16:47.863 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.35808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33206.log 2026-02-22T20:16:47.864 INFO:teuthology.orchestra.run.trial017.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.37472.log: -- replaced with /var/log/ceph/cluster1-client.admin.35808.log.gz 2026-02-22T20:16:47.864 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60579.log 2026-02-22T20:16:47.864 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37472.log.gz 2026-02-22T20:16:47.864 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.33206.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56154.log 2026-02-22T20:16:47.864 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60579.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.33206.log.gz 2026-02-22T20:16:47.865 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58558.log 2026-02-22T20:16:47.865 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60579.log.gz 2026-02-22T20:16:47.865 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.56154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26709.log 2026-02-22T20:16:47.865 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56154.log.gz 2026-02-22T20:16:47.865 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.58558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27854.log 2026-02-22T20:16:47.866 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.26709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51277.log 2026-02-22T20:16:47.866 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.26709.log.gz 2026-02-22T20:16:47.866 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.27854.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44766.log 2026-02-22T20:16:47.866 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58558.log.gz 2026-02-22T20:16:47.866 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.27854.log.gz 2026-02-22T20:16:47.866 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.51277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44574.log 2026-02-22T20:16:47.867 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27908.log 2026-02-22T20:16:47.867 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44766.log.gz 2026-02-22T20:16:47.867 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28563.log 2026-02-22T20:16:47.867 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44574.log.gz 2026-02-22T20:16:47.868 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.27908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57850.log 2026-02-22T20:16:47.868 INFO:teuthology.orchestra.run.trial017.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51277.log.gz 2026-02-22T20:16:47.868 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.28563.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.27908.log.gz 2026-02-22T20:16:47.868 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20449.log 2026-02-22T20:16:47.868 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24736.log 2026-02-22T20:16:47.868 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.28563.log.gz 2026-02-22T20:16:47.869 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.20449.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57850.log.gz 2026-02-22T20:16:47.869 INFO:teuthology.orchestra.run.trial017.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.admin.20449.log.gz 2026-02-22T20:16:47.869 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41156.log 2026-02-22T20:16:47.869 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.24736.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.24736.log.gz 2026-02-22T20:16:47.869 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.21794.log 2026-02-22T20:16:47.869 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.41156.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46111.log 2026-02-22T20:16:47.870 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.21794.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49261.log 2026-02-22T20:16:47.870 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48967.log 2026-02-22T20:16:47.870 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41156.log.gz 2026-02-22T20:16:47.870 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46111.log.gz 2026-02-22T20:16:47.870 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.49261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47958.log 2026-02-22T20:16:47.871 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.48967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29594.log 2026-02-22T20:16:47.871 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52147.log 2026-02-22T20:16:47.872 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.29594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25812.log 2026-02-22T20:16:47.872 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52147.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47958.log.gz 2026-02-22T20:16:47.872 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28120.log 2026-02-22T20:16:47.872 INFO:teuthology.orchestra.run.trial017.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52147.log.gz 91.9% 2026-02-22T20:16:47.872 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48967.log.gz 2026-02-22T20:16:47.872 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.25812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33002.log 2026-02-22T20:16:47.873 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.29594.log.gz 2026-02-22T20:16:47.873 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.28120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23606.log 2026-02-22T20:16:47.873 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.33002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26736.log 2026-02-22T20:16:47.873 INFO:teuthology.orchestra.run.trial017.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.25812.log.gz 2026-02-22T20:16:47.873 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.33002.log.gz 2026-02-22T20:16:47.874 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47826.log 2026-02-22T20:16:47.874 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23606.log: /var/log/ceph/cluster1-client.admin.26736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52809.log 2026-02-22T20:16:47.874 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.26736.log.gz 2026-02-22T20:16:47.874 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28120.log.gz 2026-02-22T20:16:47.874 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.23606.log.gz 2026-02-22T20:16:47.874 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47826.log: 83.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47826.log.gz 2026-02-22T20:16:47.874 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49535.log 2026-02-22T20:16:47.875 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52809.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32838.log 2026-02-22T20:16:47.875 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33883.log 2026-02-22T20:16:47.875 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.49535.log: /var/log/ceph/cluster1-client.admin.32838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.50615.log 2026-02-22T20:16:47.875 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.32838.log.gz 2026-02-22T20:16:47.876 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49261.log.gz 2026-02-22T20:16:47.876 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.59342.log 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.33883.log: /var/log/ceph/cluster1-client.mirror.2.50615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31421.log 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52809.log.gz 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20693.log 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.59342.log: 19.0% -- replaced with /var/log/ceph/cluster1-client.mirror.2.59342.log.gz 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33883.log.gz 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.31421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.59386.log 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.20693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46613.log 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr: 63.3% -- replaced with /var/log/ceph/cluster2-client.admin.20693.log.gz 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.3.59386.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.31421.log.gz 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr: 19.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.59386.log.gz 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47675.log 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.46613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30900.log 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46613.log.gz 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.47675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24217.log 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.30900.log: gzip 88.1% -5 --verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.47675.log.gz /var/log/ceph/cluster1-client.mirror.31454.log 2026-02-22T20:16:47.878 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.879 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24217.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27751.log 2026-02-22T20:16:47.879 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30900.log.gz 2026-02-22T20:16:47.879 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.31454.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.55408.log 2026-02-22T20:16:47.879 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.27751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38230.log 2026-02-22T20:16:47.880 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.1.55408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27827.log 2026-02-22T20:16:47.880 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.38230.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.31454.log.gz 2026-02-22T20:16:47.880 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36104.log 2026-02-22T20:16:47.880 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.24217.log.gz 2026-02-22T20:16:47.880 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.27827.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38230.log.gz 2026-02-22T20:16:47.880 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.27751.log.gz 2026-02-22T20:16:47.880 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40318.log 2026-02-22T20:16:47.880 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.27827.log.gz 2026-02-22T20:16:47.881 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.36104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47759.log 2026-02-22T20:16:47.881 INFO:teuthology.orchestra.run.trial017.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.2.50615.log.gz 2026-02-22T20:16:47.881 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26927.log 2026-02-22T20:16:47.881 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.40318.log: /var/log/ceph/cluster2-client.mirror.47759.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49535.log.gz 2026-02-22T20:16:47.881 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36104.log.gz 2026-02-22T20:16:47.882 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.40318.log.gz 2026-02-22T20:16:47.882 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58185.log 2026-02-22T20:16:47.882 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47545.log 2026-02-22T20:16:47.882 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.26927.log: /var/log/ceph/cluster1-client.mirror.58185.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47215.log 2026-02-22T20:16:47.882 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.26927.log.gz 2026-02-22T20:16:47.882 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47759.log.gz 2026-02-22T20:16:47.883 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51217.log 2026-02-22T20:16:47.883 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31368.log 2026-02-22T20:16:47.883 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47215.log: /var/log/ceph/cluster2-client.admin.51217.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47545.log.gz 2026-02-22T20:16:47.883 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54088.log 2026-02-22T20:16:47.883 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.51217.log.gz 2026-02-22T20:16:47.884 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.31368.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47215.log.gz 2026-02-22T20:16:47.884 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35735.log 2026-02-22T20:16:47.884 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.54088.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.31368.log.gz 2026-02-22T20:16:47.884 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36842.log 2026-02-22T20:16:47.884 INFO:teuthology.orchestra.run.trial017.stderr:gzip 87.1% -5 --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.54088.log.gz -- 2026-02-22T20:16:47.885 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster2-client.admin.39582.log 2026-02-22T20:16:47.885 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.35735.log: /var/log/ceph/cluster1-client.mirror.36842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52598.log 2026-02-22T20:16:47.885 INFO:teuthology.orchestra.run.trial017.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36842.log.gz 2026-02-22T20:16:47.885 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.39582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38859.log 2026-02-22T20:16:47.885 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35735.log.gz 2026-02-22T20:16:47.886 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.39582.log.gz 2026-02-22T20:16:47.886 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52598.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.55408.log.gz 2026-02-22T20:16:47.886 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37671.log 2026-02-22T20:16:47.886 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52598.log.gz 2026-02-22T20:16:47.886 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30097.log 2026-02-22T20:16:47.886 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.38859.log: /var/log/ceph/cluster1-client.admin.37671.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.38859.log.gz 2026-02-22T20:16:47.887 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38335.log 2026-02-22T20:16:47.887 INFO:teuthology.orchestra.run.trial017.stderr:gzip 87.5% -5 -- replaced with /var/log/ceph/cluster1-client.admin.37671.log.gz --verbose 2026-02-22T20:16:47.887 INFO:teuthology.orchestra.run.trial017.stderr: -- /var/log/ceph/cluster2-client.mirror.29277.log 2026-02-22T20:16:47.887 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.30097.log: /var/log/ceph/cluster2-client.admin.38335.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.38335.log.gz 2026-02-22T20:16:47.887 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19197.log 2026-02-22T20:16:47.887 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.30097.log.gz 2026-02-22T20:16:47.887 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47170.log 2026-02-22T20:16:47.888 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.29277.log: /var/log/ceph/cluster2-client.admin.19197.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19197.log.gz 2026-02-22T20:16:47.888 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35682.log 2026-02-22T20:16:47.888 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.29277.log.gz 2026-02-22T20:16:47.888 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47170.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59963.log 2026-02-22T20:16:47.888 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33830.log 2026-02-22T20:16:47.889 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.35682.log: /var/log/ceph/cluster2-client.mirror.59963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26661.log 2026-02-22T20:16:47.889 INFO:teuthology.orchestra.run.trial017.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47170.log.gz 2026-02-22T20:16:47.889 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.35682.log.gz 2026-02-22T20:16:47.889 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.33830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22281.log 2026-02-22T20:16:47.889 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.26661.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.33830.log.gz 2026-02-22T20:16:47.890 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49455.log 2026-02-22T20:16:47.890 INFO:teuthology.orchestra.run.trial017.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.26661.log.gz 2026-02-22T20:16:47.890 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.22281.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27005.log 2026-02-22T20:16:47.890 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.49455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46676.log 2026-02-22T20:16:47.890 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.22281.log.gz 2026-02-22T20:16:47.891 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.27005.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.49455.log.gz 2026-02-22T20:16:47.891 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55204.log 2026-02-22T20:16:47.891 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.27005.log.gz 2026-02-22T20:16:47.891 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46089.log 2026-02-22T20:16:47.891 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.55204.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46676.log.gz 2026-02-22T20:16:47.891 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51320.log 2026-02-22T20:16:47.891 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.55204.log.gz 2026-02-22T20:16:47.892 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46089.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24598.log 2026-02-22T20:16:47.892 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.51320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54592.log 2026-02-22T20:16:47.892 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38491.log 2026-02-22T20:16:47.892 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24598.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46089.log.gz 2026-02-22T20:16:47.893 INFO:teuthology.orchestra.run.trial017.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51320.log.gz 2026-02-22T20:16:47.893 INFO:teuthology.orchestra.run.trial017.stderr: 67.5% -- replaced with /var/log/ceph/cluster2-client.mirror.24598.log.gz 2026-02-22T20:16:47.893 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.54592.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59963.log.gz 2026-02-22T20:16:47.893 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51005.log 2026-02-22T20:16:47.893 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54592.log.gz 2026-02-22T20:16:47.893 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.38491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44262.log 2026-02-22T20:16:47.893 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.38491.log.gz 2026-02-22T20:16:47.893 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.51005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59986.log 2026-02-22T20:16:47.894 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22734.log 2026-02-22T20:16:47.894 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44262.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.51005.log.gz 2026-02-22T20:16:47.894 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.59986.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.51682.log 2026-02-22T20:16:47.894 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.22734.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59986.log.gz 2026-02-22T20:16:47.894 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44262.log.gz 2026-02-22T20:16:47.894 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53208.log 2026-02-22T20:16:47.894 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.22734.log.gz 2026-02-22T20:16:47.895 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.3.51682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42313.log 2026-02-22T20:16:47.895 INFO:teuthology.orchestra.run.trial017.stderr: 20.2% -- replaced with /var/log/ceph/cluster1-client.mirror.3.51682.log.gz 2026-02-22T20:16:47.895 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42751.log 2026-02-22T20:16:47.895 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.53208.log: /var/log/ceph/cluster1-client.mirror.42313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57216.log 2026-02-22T20:16:47.895 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42313.log.gz 2026-02-22T20:16:47.896 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53208.log.gz 2026-02-22T20:16:47.896 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.42751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58293.log 2026-02-22T20:16:47.896 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.48309.log 2026-02-22T20:16:47.896 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57216.log.gz 2026-02-22T20:16:47.896 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.58293.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.42751.log.gz 2026-02-22T20:16:47.897 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29882.log 2026-02-22T20:16:47.897 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.58293.log.gz 2026-02-22T20:16:47.897 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.48309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35781.log 2026-02-22T20:16:47.897 INFO:teuthology.orchestra.run.trial017.stderr: 20.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.48309.log.gz 2026-02-22T20:16:47.897 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.29882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40003.log 2026-02-22T20:16:47.897 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34973.log 2026-02-22T20:16:47.898 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.29882.log.gz 2026-02-22T20:16:47.898 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.35781.log: /var/log/ceph/cluster2-client.mirror.40003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49130.log 2026-02-22T20:16:47.898 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.35781.log.gz 2026-02-22T20:16:47.898 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40003.log.gz 2026-02-22T20:16:47.898 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.34973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49241.log 2026-02-22T20:16:47.899 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.34973.log.gz 2026-02-22T20:16:47.899 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.49130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26187.log 2026-02-22T20:16:47.899 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.49241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33407.log 2026-02-22T20:16:47.899 INFO:teuthology.orchestra.run.trial017.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.admin.49130.log.gz 2026-02-22T20:16:47.899 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.26187.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49241.log.gz 2026-02-22T20:16:47.899 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37940.log 2026-02-22T20:16:47.900 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.33407.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.26187.log.gz 2026-02-22T20:16:47.900 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57898.log 2026-02-22T20:16:47.900 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.33407.log.gz 2026-02-22T20:16:47.900 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40838.log 2026-02-22T20:16:47.900 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.37940.log: /var/log/ceph/cluster1-client.mirror.57898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38778.log 2026-02-22T20:16:47.900 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57898.log.gz 2026-02-22T20:16:47.901 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.37940.log.gz 2026-02-22T20:16:47.901 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.40838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35580.log 2026-02-22T20:16:47.901 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.38778.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40838.log.gz 2026-02-22T20:16:47.901 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.38778.log.gz 2026-02-22T20:16:47.901 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30769.log 2026-02-22T20:16:47.901 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.35580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37421.log 2026-02-22T20:16:47.902 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.35580.log.gz 2026-02-22T20:16:47.902 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.30769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45167.log 2026-02-22T20:16:47.902 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.30769.log.gz 2026-02-22T20:16:47.902 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.37421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19355.log 2026-02-22T20:16:47.903 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58368.log 2026-02-22T20:16:47.903 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57752.log 2026-02-22T20:16:47.903 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19355.log.gz 2026-02-22T20:16:47.903 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37421.log.gz 2026-02-22T20:16:47.903 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.58368.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45167.log.gz 2026-02-22T20:16:47.903 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41681.log 2026-02-22T20:16:47.904 INFO:teuthology.orchestra.run.trial017.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.58368.log.gz 2026-02-22T20:16:47.904 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20061.log 2026-02-22T20:16:47.904 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57752.log.gz 2026-02-22T20:16:47.904 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.41681.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41070.log 2026-02-22T20:16:47.904 INFO:teuthology.orchestra.run.trial017.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.41681.log.gz 2026-02-22T20:16:47.905 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20037.log 2026-02-22T20:16:47.905 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.20061.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20061.log.gz 2026-02-22T20:16:47.905 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.41070.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.57362.log 2026-02-22T20:16:47.905 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.41070.log.gz 2026-02-22T20:16:47.905 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.20037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54454.log 2026-02-22T20:16:47.905 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20037.log.gz 2026-02-22T20:16:47.905 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.57362.log: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.57362.log.gz 2026-02-22T20:16:47.906 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34437.log 2026-02-22T20:16:47.906 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.54454.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50933.log 2026-02-22T20:16:47.906 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54454.log.gz 2026-02-22T20:16:47.906 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22616.log 2026-02-22T20:16:47.906 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.34437.log: /var/log/ceph/cluster2-client.admin.50933.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32154.log 2026-02-22T20:16:47.907 INFO:teuthology.orchestra.run.trial017.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.admin.50933.log.gz 2026-02-22T20:16:47.907 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.22616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37080.log 2026-02-22T20:16:47.907 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27519.log 2026-02-22T20:16:47.907 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.32154.log: /var/log/ceph/cluster1-client.mirror.37080.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.34437.log.gz 2026-02-22T20:16:47.907 INFO:teuthology.orchestra.run.trial017.stderr: 87.6%gzip -- replaced with /var/log/ceph/cluster2-client.admin.32154.log.gz 2026-02-22T20:16:47.907 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20535.log 2026-02-22T20:16:47.907 INFO:teuthology.orchestra.run.trial017.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.22616.log.gz 2026-02-22T20:16:47.908 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37080.log.gz 2026-02-22T20:16:47.908 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.27519.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60527.log 2026-02-22T20:16:47.908 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.20535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39528.log 2026-02-22T20:16:47.908 INFO:teuthology.orchestra.run.trial017.stderr: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.20535.log.gz 2026-02-22T20:16:47.908 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60527.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60527.log.gz 2026-02-22T20:16:47.909 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40165.log 2026-02-22T20:16:47.909 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.39528.log: -- /var/log/ceph/cluster2-client.admin.26790.log 2026-02-22T20:16:47.909 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.40165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.48448.log 2026-02-22T20:16:47.910 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.39528.log.gz 2026-02-22T20:16:47.910 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.40165.log.gz 2026-02-22T20:16:47.910 INFO:teuthology.orchestra.run.trial017.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27519.log.gz 2026-02-22T20:16:47.910 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.26790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45643.log 2026-02-22T20:16:47.910 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.26790.log.gz 2026-02-22T20:16:47.910 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.3.48448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55042.log 2026-02-22T20:16:47.910 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26558.log 2026-02-22T20:16:47.910 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45081.log 2026-02-22T20:16:47.910 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.55042.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45643.log.gz 2026-02-22T20:16:47.911 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.26558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52524.log 2026-02-22T20:16:47.911 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45081.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.26558.log.gz 2026-02-22T20:16:47.911 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55228.log 2026-02-22T20:16:47.911 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.52524.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45081.log.gz 2026-02-22T20:16:47.911 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55042.log.gz 2026-02-22T20:16:47.911 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53410.log 2026-02-22T20:16:47.912 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52524.log.gz 2026-02-22T20:16:47.912 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.55228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38437.log 2026-02-22T20:16:47.912 INFO:teuthology.orchestra.run.trial017.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.48448.log.gz 2026-02-22T20:16:47.912 INFO:teuthology.orchestra.run.trial017.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.55228.log.gz 2026-02-22T20:16:47.912 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.53410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36916.log 2026-02-22T20:16:47.912 INFO:teuthology.orchestra.run.trial017.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53410.log.gz 2026-02-22T20:16:47.912 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.50571.log 2026-02-22T20:16:47.913 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.38437.log: /var/log/ceph/cluster1-client.admin.36916.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.38437.log.gz 2026-02-22T20:16:47.913 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45242.log 2026-02-22T20:16:47.913 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.36916.log.gz 2026-02-22T20:16:47.913 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.50571.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48880.log 2026-02-22T20:16:47.913 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.45242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20484.log 2026-02-22T20:16:47.913 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45242.log.gz 2026-02-22T20:16:47.913 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.48880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28590.log 2026-02-22T20:16:47.914 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.20484.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.admin.48880.log.gz 2026-02-22T20:16:47.914 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48714.log 2026-02-22T20:16:47.914 INFO:teuthology.orchestra.run.trial017.stderr: 56.6% -- replaced with /var/log/ceph/cluster2-client.mirror.20484.log.gz 2026-02-22T20:16:47.914 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.28590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.57348.log 2026-02-22T20:16:47.915 INFO:teuthology.orchestra.run.trial017.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.50571.log.gz 2026-02-22T20:16:47.915 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.28590.log.gz 2026-02-22T20:16:47.915 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40291.log 2026-02-22T20:16:47.915 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.48714.log: /var/log/ceph/cluster1-client.mirror.0.57348.log: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.57348.log.gz 2026-02-22T20:16:47.915 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34010.log 2026-02-22T20:16:47.915 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48714.log.gz 2026-02-22T20:16:47.915 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.40291.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40291.log.gz 2026-02-22T20:16:47.915 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31895.log 2026-02-22T20:16:47.916 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.34010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21274.log 2026-02-22T20:16:47.916 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.31895.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.34010.log.gz 2026-02-22T20:16:47.916 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29696.log 2026-02-22T20:16:47.916 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.31895.log.gz 2026-02-22T20:16:47.916 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.21274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30530.log 2026-02-22T20:16:47.916 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.29696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38206.log 2026-02-22T20:16:47.916 INFO:teuthology.orchestra.run.trial017.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.21274.log.gz 2026-02-22T20:16:47.916 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.29696.log.gz 2026-02-22T20:16:47.917 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.30530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45341.log 2026-02-22T20:16:47.917 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.38206.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31799.log 2026-02-22T20:16:47.917 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38206.log.gz 2026-02-22T20:16:47.917 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.45341.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26161.log 2026-02-22T20:16:47.917 INFO:teuthology.orchestra.run.trial017.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30530.log.gz 2026-02-22T20:16:47.918 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.31799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40586.log 2026-02-22T20:16:47.918 INFO:teuthology.orchestra.run.trial017.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.45341.log.gz 2026-02-22T20:16:47.918 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31799.log.gz 2026-02-22T20:16:47.918 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.26161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37448.log 2026-02-22T20:16:47.918 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.26161.log.gz 2026-02-22T20:16:47.918 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.40586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26004.log 2026-02-22T20:16:47.919 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.51653.log 2026-02-22T20:16:47.919 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.37448.log: /var/log/ceph/cluster2-client.mirror.26004.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40586.log.gz 2026-02-22T20:16:47.919 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49428.log 2026-02-22T20:16:47.919 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.37448.log.gz 2026-02-22T20:16:47.919 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.2.51653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47082.log 2026-02-22T20:16:47.919 INFO:teuthology.orchestra.run.trial017.stderr: 20.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.51653.log.gz 2026-02-22T20:16:47.919 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.49428.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.26004.log.gz 2026-02-22T20:16:47.919 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49945.log 2026-02-22T20:16:47.920 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.49428.log.gz 2026-02-22T20:16:47.920 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.21454.log 2026-02-22T20:16:47.920 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.47082.log: /var/log/ceph/cluster1-client.mirror.49945.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47082.log.gz 2026-02-22T20:16:47.920 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40036.log 2026-02-22T20:16:47.920 INFO:teuthology.orchestra.run.trial017.stderr: 96.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.57385.log.gz 2026-02-22T20:16:47.920 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.21454.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41097.log 2026-02-22T20:16:47.921 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34036.log 2026-02-22T20:16:47.921 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.21454.log.gz 2026-02-22T20:16:47.921 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.40036.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49945.log.gz 2026-02-22T20:16:47.921 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.41097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37725.log 2026-02-22T20:16:47.921 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.41097.log.gz 2026-02-22T20:16:47.922 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.34036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45729.log 2026-02-22T20:16:47.922 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47129.log 2026-02-22T20:16:47.922 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.37725.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40036.log.gz 2026-02-22T20:16:47.922 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.45729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42551.log 2026-02-22T20:16:47.922 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.37725.log.gz 2026-02-22T20:16:47.922 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45729.log.gz 2026-02-22T20:16:47.923 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34036.log.gz 2026-02-22T20:16:47.923 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47985.log 2026-02-22T20:16:47.923 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.42551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46050.log 2026-02-22T20:16:47.923 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47985.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47129.log.gz 2026-02-22T20:16:47.923 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56333.log 2026-02-22T20:16:47.924 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47985.log.gz 2026-02-22T20:16:47.924 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25096.log 2026-02-22T20:16:47.924 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.46050.log: /var/log/ceph/cluster2-client.mirror.56333.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49356.log 2026-02-22T20:16:47.924 INFO:teuthology.orchestra.run.trial017.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46050.log.gz 2026-02-22T20:16:47.924 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.25096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35072.log 2026-02-22T20:16:47.925 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.25096.log.gz 2026-02-22T20:16:47.925 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24330.log 2026-02-22T20:16:47.925 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.49356.log: /var/log/ceph/cluster1-client.admin.35072.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.21794.log.gz 2026-02-22T20:16:47.925 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49356.log.gz 2026-02-22T20:16:47.925 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41043.log 2026-02-22T20:16:47.925 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.35072.log.gz 2026-02-22T20:16:47.925 INFO:teuthology.orchestra.run.trial017.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58185.log.gz 2026-02-22T20:16:47.926 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.24330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.51620.log 2026-02-22T20:16:47.926 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.41043.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34069.log 2026-02-22T20:16:47.926 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.24330.log.gz 2026-02-22T20:16:47.926 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.0.51620.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52855.log 2026-02-22T20:16:47.926 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.41043.log.gz 2026-02-22T20:16:47.926 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.34069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40659.log 2026-02-22T20:16:47.927 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26080.log 2026-02-22T20:16:47.927 INFO:teuthology.orchestra.run.trial017.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52855.log.gz 2026-02-22T20:16:47.927 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.40659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41996.log 2026-02-22T20:16:47.928 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.26080.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.40659.log.gz 2026-02-22T20:16:47.928 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29723.log 2026-02-22T20:16:47.928 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34069.log.gz 2026-02-22T20:16:47.928 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.26080.log.gz 2026-02-22T20:16:47.928 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32765.log 2026-02-22T20:16:47.928 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.41996.log: /var/log/ceph/cluster1-client.admin.29723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24811.log 2026-02-22T20:16:47.928 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.41996.log.gz 2026-02-22T20:16:47.928 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.29723.log.gz 2026-02-22T20:16:47.928 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53307.log 2026-02-22T20:16:47.929 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.32765.log: /var/log/ceph/cluster1-client.mirror.24811.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.24811.log.gz 2026-02-22T20:16:47.929 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56130.log 2026-02-22T20:16:47.929 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42676.log 2026-02-22T20:16:47.929 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.53307.log: /var/log/ceph/cluster2-client.mirror.56130.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42551.log.gz 2026-02-22T20:16:47.929 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32765.log.gz 2026-02-22T20:16:47.929 INFO:teuthology.orchestra.run.trial017.stderr: 87.8%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.56130.log.gz 2026-02-22T20:16:47.929 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.54857.log 2026-02-22T20:16:47.929 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.53307.log.gz 2026-02-22T20:16:47.930 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.42676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48138.log 2026-02-22T20:16:47.930 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42676.log.gz 2026-02-22T20:16:47.930 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.54857.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39375.log 2026-02-22T20:16:47.930 INFO:teuthology.orchestra.run.trial017.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.admin.54857.log.gz 2026-02-22T20:16:47.930 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.48138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38885.log 2026-02-22T20:16:47.931 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.39375.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22440.log 2026-02-22T20:16:47.931 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.admin.48138.log.gz 2026-02-22T20:16:47.931 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.38885.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.39375.log.gz 2026-02-22T20:16:47.931 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56226.log 2026-02-22T20:16:47.931 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49600.log 2026-02-22T20:16:47.932 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.22440.log: /var/log/ceph/cluster1-client.admin.56226.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.56226.log.gz 2026-02-22T20:16:47.932 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.57357.log 2026-02-22T20:16:47.932 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38885.log.gz 2026-02-22T20:16:47.932 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43986.log 2026-02-22T20:16:47.932 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.49600.log: /var/log/ceph/cluster1-client.mirror.0.57357.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.49600.log.gz 2026-02-22T20:16:47.932 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.55385.log 2026-02-22T20:16:47.933 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43986.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40264.log 2026-02-22T20:16:47.933 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.1.55385.log: 19.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.55385.log.gz 2026-02-22T20:16:47.933 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58857.log 2026-02-22T20:16:47.933 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43986.log.gz 2026-02-22T20:16:47.933 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.40264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47716.log 2026-02-22T20:16:47.934 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55601.log 2026-02-22T20:16:47.934 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.58857.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40264.log.gz 2026-02-22T20:16:47.934 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46177.log 2026-02-22T20:16:47.934 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.58857.log.gz 2026-02-22T20:16:47.934 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47716.log.gz 2026-02-22T20:16:47.935 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.55601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39824.log 2026-02-22T20:16:47.935 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55601.log.gz 2026-02-22T20:16:47.935 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46177.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45685.log 2026-02-22T20:16:47.935 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.39824.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46177.log.gz 2026-02-22T20:16:47.935 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.59314.log 2026-02-22T20:16:47.935 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.39824.log.gz 2026-02-22T20:16:47.936 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31670.log 2026-02-22T20:16:47.936 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.59314.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50052.log 2026-02-22T20:16:47.936 INFO:teuthology.orchestra.run.trial017.stderr: 19.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.59314.log.gz 2026-02-22T20:16:47.936 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40111.log 2026-02-22T20:16:47.937 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.31670.log: /var/log/ceph/cluster2-client.mirror.50052.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20170.log 2026-02-22T20:16:47.937 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.31670.log.gz 2026-02-22T20:16:47.937 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50052.log.gz 2026-02-22T20:16:47.937 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.40111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53430.log 2026-02-22T20:16:47.937 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.20170.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20170.log.gz 2026-02-22T20:16:47.937 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27328.log 2026-02-22T20:16:47.937 INFO:teuthology.orchestra.run.trial017.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.40111.log.gz 2026-02-22T20:16:47.938 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.53430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20362.log 2026-02-22T20:16:47.938 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.22440.log.gz 2026-02-22T20:16:47.938 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.27328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54533.log 2026-02-22T20:16:47.939 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.55457.log 2026-02-22T20:16:47.939 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.20362.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53430.log.gz 2026-02-22T20:16:47.939 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.54533.log: 57.6% -- replaced with /var/log/ceph/cluster1-client.admin.20362.log.gz 2026-02-22T20:16:47.939 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54533.log.gz 2026-02-22T20:16:47.939 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58953.log 2026-02-22T20:16:47.939 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.3.55457.log: gzip 19.6% -- replaced with /var/log/ceph/cluster2-client.mirror.3.55457.log.gz 2026-02-22T20:16:47.939 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47479.log 2026-02-22T20:16:47.939 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.58953.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.27328.log.gz 2026-02-22T20:16:47.939 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42076.log 2026-02-22T20:16:47.940 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.47479.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39609.log 2026-02-22T20:16:47.940 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19555.log 2026-02-22T20:16:47.940 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.42076.log: /var/log/ceph/cluster2-client.admin.39609.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47479.log.gz 2026-02-22T20:16:47.940 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55108.log 2026-02-22T20:16:47.941 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.39609.log.gz 2026-02-22T20:16:47.941 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19555.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19555.log.gz 2026-02-22T20:16:47.941 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57874.log 2026-02-22T20:16:47.941 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.55108.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42076.log.gz 2026-02-22T20:16:47.941 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35494.log 2026-02-22T20:16:47.941 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.55108.log.gz 2026-02-22T20:16:47.942 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.57874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58731.log 2026-02-22T20:16:47.942 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57874.log.gz 2026-02-22T20:16:47.942 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52875.log 2026-02-22T20:16:47.942 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.35494.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45685.log.gz 2026-02-22T20:16:47.942 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.58731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46379.log 2026-02-22T20:16:47.942 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.35494.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.58731.log.gz 2026-02-22T20:16:47.942 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:47.943 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.52875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37698.log 2026-02-22T20:16:47.943 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31242.log 2026-02-22T20:16:47.943 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38577.log 2026-02-22T20:16:47.943 INFO:teuthology.orchestra.run.trial017.stderr: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46379.log.gz 2026-02-22T20:16:47.944 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.37698.log: /var/log/ceph/cluster1-client.admin.31242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36446.log 2026-02-22T20:16:47.944 INFO:teuthology.orchestra.run.trial017.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.37698.log.gz 2026-02-22T20:16:47.944 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.31242.log.gz 2026-02-22T20:16:47.944 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.38577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56202.log 2026-02-22T20:16:47.944 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38577.log.gz 2026-02-22T20:16:47.944 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.36446.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52875.log.gz 2026-02-22T20:16:47.944 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.50548.log 2026-02-22T20:16:47.944 INFO:teuthology.orchestra.run.trial017.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.36446.log.gz 2026-02-22T20:16:47.945 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.56202.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22224.log 2026-02-22T20:16:47.945 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56202.log.gz 2026-02-22T20:16:47.945 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.50548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37185.log 2026-02-22T20:16:47.945 INFO:teuthology.orchestra.run.trial017.stderr: 20.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.50548.log.gz 2026-02-22T20:16:47.945 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.22224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23940.log 2026-02-22T20:16:47.946 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.22224.log.gz 2026-02-22T20:16:47.946 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21373.log 2026-02-22T20:16:47.946 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.37185.log: /var/log/ceph/cluster1-client.admin.23940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.21843.log 2026-02-22T20:16:47.946 INFO:teuthology.orchestra.run.trial017.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.37185.log.gz 2026-02-22T20:16:47.946 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.21373.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.23940.log.gz 2026-02-22T20:16:47.947 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34324.log 2026-02-22T20:16:47.947 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.21373.log.gz 2026-02-22T20:16:47.947 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.3.21843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34999.log 2026-02-22T20:16:47.947 INFO:teuthology.orchestra.run.trial017.stderr: 19.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.21843.log.gz 2026-02-22T20:16:47.947 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.34324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29408.log 2026-02-22T20:16:47.948 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.34999.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.34324.log.gz 2026-02-22T20:16:47.948 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56038.log 2026-02-22T20:16:47.948 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.29408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59125.log 2026-02-22T20:16:47.948 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34999.log.gz 2026-02-22T20:16:47.948 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43038.log 2026-02-22T20:16:47.949 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.56038.log: /var/log/ceph/cluster2-client.mirror.59125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38706.log 2026-02-22T20:16:47.949 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.43038.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29408.log.gz 2026-02-22T20:16:47.949 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30223.log 2026-02-22T20:16:47.949 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43038.log.gz 2026-02-22T20:16:47.949 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.38706.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59125.log.gz 2026-02-22T20:16:47.949 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60397.log 2026-02-22T20:16:47.949 INFO:teuthology.orchestra.run.trial017.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38706.log.gz 2026-02-22T20:16:47.950 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.30223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26395.log 2026-02-22T20:16:47.950 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.60397.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60397.log.gz 2026-02-22T20:16:47.950 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.30223.log.gz 2026-02-22T20:16:47.950 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29669.log 2026-02-22T20:16:47.950 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.26395.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29935.log 2026-02-22T20:16:47.951 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.29669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16310.log 2026-02-22T20:16:47.951 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26395.log.gz 2026-02-22T20:16:47.951 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.29669.log.gz 2026-02-22T20:16:47.951 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.29935.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29776.log 2026-02-22T20:16:47.951 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.16310.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16310.log.gz 2026-02-22T20:16:47.951 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32581.log 2026-02-22T20:16:47.952 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.29776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34619.log 2026-02-22T20:16:47.952 INFO:teuthology.orchestra.run.trial017.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.29935.log.gz 2026-02-22T20:16:47.952 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.32581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46546.log 2026-02-22T20:16:47.952 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.34619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46777.log 2026-02-22T20:16:47.952 INFO:teuthology.orchestra.run.trial017.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29776.log.gz 2026-02-22T20:16:47.953 INFO:teuthology.orchestra.run.trial017.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32581.log.gz 2026-02-22T20:16:47.953 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53457.log 2026-02-22T20:16:47.953 INFO:teuthology.orchestra.run.trial017.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34619.log.gz 2026-02-22T20:16:47.953 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39896.log 2026-02-22T20:16:47.953 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.53457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20586.log 2026-02-22T20:16:47.954 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.39896.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46777.log.gz 2026-02-22T20:16:47.954 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53457.log.gz 2026-02-22T20:16:47.954 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46925.log 2026-02-22T20:16:47.954 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.39896.log.gz 2026-02-22T20:16:47.954 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33587.log 2026-02-22T20:16:47.954 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.20586.log: 58.7% -- replaced with /var/log/ceph/cluster2-client.mirror.20586.log.gz 2026-02-22T20:16:47.955 INFO:teuthology.orchestra.run.trial017.stderr:gzip/var/log/ceph/cluster2-client.mirror.46925.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53113.log 2026-02-22T20:16:47.955 INFO:teuthology.orchestra.run.trial017.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56038.log.gz 2026-02-22T20:16:47.955 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.33587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46525.log 2026-02-22T20:16:47.955 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.53113.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.33587.log.gz 2026-02-22T20:16:47.955 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53602.log 2026-02-22T20:16:47.955 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53113.log.gz 2026-02-22T20:16:47.955 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46461.log 2026-02-22T20:16:47.956 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.53602.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46525.log.gz 2026-02-22T20:16:47.956 INFO:teuthology.orchestra.run.trial017.stderr:gzip 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.53602.log.gz 2026-02-22T20:16:47.956 INFO:teuthology.orchestra.run.trial017.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24262.log 2026-02-22T20:16:47.956 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43448.log 2026-02-22T20:16:47.956 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24262.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46925.log.gz 2026-02-22T20:16:47.956 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19347.log 2026-02-22T20:16:47.957 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.43448.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46461.log.gz 2026-02-22T20:16:47.957 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43448.log.gz 2026-02-22T20:16:47.957 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60119.log 2026-02-22T20:16:47.957 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45774.log 2026-02-22T20:16:47.957 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19347.log.gz 2026-02-22T20:16:47.957 INFO:teuthology.orchestra.run.trial017.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.24262.log.gz 2026-02-22T20:16:47.958 INFO:teuthology.orchestra.run.trial017.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46546.log.gz 2026-02-22T20:16:47.958 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.60119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44381.log 2026-02-22T20:16:47.958 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.60119.log.gz 2026-02-22T20:16:47.958 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28195.log 2026-02-22T20:16:47.958 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.45774.log: /var/log/ceph/cluster1-client.mirror.44381.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45774.log.gz 2026-02-22T20:16:47.958 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43965.log 2026-02-22T20:16:47.958 INFO:teuthology.orchestra.run.trial017.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44381.log.gz 2026-02-22T20:16:47.959 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30659.log 2026-02-22T20:16:47.959 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.28195.log: /var/log/ceph/cluster2-client.mirror.43965.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28644.log 2026-02-22T20:16:47.959 INFO:teuthology.orchestra.run.trial017.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43965.log.gz 2026-02-22T20:16:47.959 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.28195.log.gz 2026-02-22T20:16:47.959 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.30659.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29621.log 2026-02-22T20:16:47.959 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.30659.log.gz 2026-02-22T20:16:47.960 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.28644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23373.log 2026-02-22T20:16:47.960 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.29621.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.28644.log.gz 2026-02-22T20:16:47.960 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.29621.log.gz 2026-02-22T20:16:47.960 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50168.log 2026-02-22T20:16:47.960 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49751.log 2026-02-22T20:16:47.961 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23373.log: /var/log/ceph/cluster2-client.mirror.50168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54067.log 2026-02-22T20:16:47.961 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.23373.log.gz 2026-02-22T20:16:47.961 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50168.log.gz 2026-02-22T20:16:47.961 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.49751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34251.log 2026-02-22T20:16:47.961 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.54067.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19198.log 2026-02-22T20:16:47.962 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.34251.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49751.log.gz 2026-02-22T20:16:47.962 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54067.log.gz 2026-02-22T20:16:47.962 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49557.log 2026-02-22T20:16:47.962 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19198.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34251.log.gz 2026-02-22T20:16:47.962 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27980.log 2026-02-22T20:16:47.962 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19198.log.gz 2026-02-22T20:16:47.963 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.49557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36365.log 2026-02-22T20:16:47.963 INFO:teuthology.orchestra.run.trial017.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.51620.log.gz 2026-02-22T20:16:47.963 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.27980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.25298.log 2026-02-22T20:16:47.963 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.36365.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42023.log 87.7% 2026-02-22T20:16:47.963 INFO:teuthology.orchestra.run.trial017.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.27980.log.gz 2026-02-22T20:16:47.963 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.2.25298.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49557.log.gz 2026-02-22T20:16:47.964 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 88.6% --verbose -- replaced with /var/log/ceph/cluster2-client.admin.36365.log.gz -- 2026-02-22T20:16:47.964 INFO:teuthology.orchestra.run.trial017.stderr: /var/log/ceph/cluster1-client.admin.33560.log 2026-02-22T20:16:47.964 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.42023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24691.log 2026-02-22T20:16:47.964 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.33560.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.42023.log.gz 2026-02-22T20:16:47.965 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33614.log 2026-02-22T20:16:47.965 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.24691.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.33560.log.gz 2026-02-22T20:16:47.965 INFO:teuthology.orchestra.run.trial017.stderr:gzip 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.24691.log.gz -5 2026-02-22T20:16:47.965 INFO:teuthology.orchestra.run.trial017.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.1.54123.log 2026-02-22T20:16:47.965 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.33614.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45004.log 2026-02-22T20:16:47.966 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.33614.log.gz 2026-02-22T20:16:47.966 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.1.54123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44983.log 2026-02-22T20:16:47.966 INFO:teuthology.orchestra.run.trial017.stderr: 19.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.54123.log.gz 2026-02-22T20:16:47.966 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39635.log 2026-02-22T20:16:47.966 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45004.log.gz 2026-02-22T20:16:47.966 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39327.log 2026-02-22T20:16:47.967 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.39635.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44983.log.gz 2026-02-22T20:16:47.967 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44683.log 2026-02-22T20:16:47.967 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.39327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44895.log 2026-02-22T20:16:47.967 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39327.log.gz 2026-02-22T20:16:47.968 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.44683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25890.log 2026-02-22T20:16:47.968 INFO:teuthology.orchestra.run.trial017.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.39635.log.gz 2026-02-22T20:16:47.968 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46992.log 2026-02-22T20:16:47.968 INFO:teuthology.orchestra.run.trial017.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44683.log.gz 2026-02-22T20:16:47.968 INFO:teuthology.orchestra.run.trial017.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44895.log.gz 2026-02-22T20:16:47.968 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.25890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27718.log 2026-02-22T20:16:47.969 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.46992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22172.log 2026-02-22T20:16:47.969 INFO:teuthology.orchestra.run.trial017.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.25890.log.gz 2026-02-22T20:16:47.969 INFO:teuthology.orchestra.run.trial017.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46992.log.gz 2026-02-22T20:16:47.969 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.27718.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45834.log 2026-02-22T20:16:47.969 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.22172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43231.log 2026-02-22T20:16:47.970 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.45834.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.22172.log.gz 2026-02-22T20:16:47.970 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47738.log 2026-02-22T20:16:47.970 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.27718.log.gz 2026-02-22T20:16:47.970 INFO:teuthology.orchestra.run.trial017.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.57357.log.gz 2026-02-22T20:16:47.970 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.55371.log 2026-02-22T20:16:47.970 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.43231.log: /var/log/ceph/cluster2-client.mirror.47738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43666.log 2026-02-22T20:16:47.971 INFO:teuthology.orchestra.run.trial017.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45834.log.gz 2026-02-22T20:16:47.971 INFO:teuthology.orchestra.run.trial017.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47738.log.gz 2026-02-22T20:16:47.971 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.0.55371.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43231.log.gz 2026-02-22T20:16:47.971 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31746.log 2026-02-22T20:16:47.971 INFO:teuthology.orchestra.run.trial017.stderr: 19.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.55371.log.gz 2026-02-22T20:16:47.971 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.43666.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44304.log 2026-02-22T20:16:47.972 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46262.log 2026-02-22T20:16:47.972 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.31746.log: /var/log/ceph/cluster2-client.mirror.44304.log: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43666.log.gz 2026-02-22T20:16:47.972 INFO:teuthology.orchestra.run.trial017.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.31746.log.gz 2026-02-22T20:16:47.972 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42652.log 2026-02-22T20:16:47.972 INFO:teuthology.orchestra.run.trial017.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44304.log.gz 2026-02-22T20:16:47.972 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50321.log 2026-02-22T20:16:47.973 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.46262.log: /var/log/ceph/cluster2-client.mirror.42652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38601.log 2026-02-22T20:16:47.973 INFO:teuthology.orchestra.run.trial017.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42652.log.gz 2026-02-22T20:16:47.973 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.50321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44745.log 2026-02-22T20:16:47.973 INFO:teuthology.orchestra.run.trial017.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46262.log.gz 2026-02-22T20:16:47.973 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32685.log 2026-02-22T20:16:47.973 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.mirror.38601.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.admin.50321.log.gz 2026-02-22T20:16:47.974 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.44745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42050.log 2026-02-22T20:16:47.974 INFO:teuthology.orchestra.run.trial017.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38601.log.gz 2026-02-22T20:16:47.974 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.32685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19432.log 2026-02-22T20:16:47.974 INFO:teuthology.orchestra.run.trial017.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.32685.log.gz 2026-02-22T20:16:47.974 INFO:teuthology.orchestra.run.trial017.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44745.log.gz 2026-02-22T20:16:47.974 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.42050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42724.log 2026-02-22T20:16:47.975 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.admin.19432.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.42050.log.gz 2026-02-22T20:16:47.975 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29481.log 2026-02-22T20:16:47.975 INFO:teuthology.orchestra.run.trial017.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19432.log.gz 2026-02-22T20:16:47.975 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.42724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32026.log 2026-02-22T20:16:47.975 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.29481.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.42724.log.gz 2026-02-22T20:16:47.975 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20569.log 2026-02-22T20:16:47.976 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.32026.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.29481.log.gz 2026-02-22T20:16:47.976 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23346.log 2026-02-22T20:16:47.976 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.20569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21189.log 2026-02-22T20:16:47.976 INFO:teuthology.orchestra.run.trial017.stderr: 58.7% -- replaced with /var/log/ceph/cluster2-client.mirror.20569.log.gz 2026-02-22T20:16:47.977 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster1-client.admin.23346.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32026.log.gz 2026-02-22T20:16:47.977 INFO:teuthology.orchestra.run.trial017.stderr:/var/log/ceph/cluster2-client.mirror.21189.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.23346.log.gz 2026-02-22T20:16:47.978 INFO:teuthology.orchestra.run.trial017.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.21189.log.gz 2026-02-22T20:16:47.981 INFO:teuthology.orchestra.run.trial017.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.25298.log.gz 2026-02-22T20:16:48.013 INFO:teuthology.orchestra.run.trial017.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56333.log.gz 2026-02-22T20:16:48.020 INFO:teuthology.orchestra.run.trial017.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58953.log.gz 2026-02-22T20:16:48.068 INFO:teuthology.orchestra.run.trial007.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2026-02-22T20:16:48.149 INFO:teuthology.orchestra.run.trial017.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2026-02-22T20:16:50.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2026-02-22T20:16:52.032 INFO:teuthology.orchestra.run.trial007.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2026-02-22T20:16:52.604 INFO:teuthology.orchestra.run.trial007.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2026-02-22T20:16:53.187 INFO:teuthology.orchestra.run.trial007.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2026-02-22T20:16:53.188 INFO:teuthology.orchestra.run.trial007.stderr: 2026-02-22T20:16:53.188 INFO:teuthology.orchestra.run.trial007.stderr:real 0m5.722s 2026-02-22T20:16:53.188 INFO:teuthology.orchestra.run.trial007.stderr:user 0m15.516s 2026-02-22T20:16:53.188 INFO:teuthology.orchestra.run.trial007.stderr:sys 0m0.701s 2026-02-22T20:16:53.679 INFO:teuthology.orchestra.run.trial017.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2026-02-22T20:16:54.441 INFO:teuthology.orchestra.run.trial017.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2026-02-22T20:16:55.115 INFO:teuthology.orchestra.run.trial017.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2026-02-22T20:16:55.116 INFO:teuthology.orchestra.run.trial017.stderr: 2026-02-22T20:16:55.117 INFO:teuthology.orchestra.run.trial017.stderr:real 0m7.644s 2026-02-22T20:16:55.117 INFO:teuthology.orchestra.run.trial017.stderr:user 0m22.300s 2026-02-22T20:16:55.117 INFO:teuthology.orchestra.run.trial017.stderr:sys 0m1.385s 2026-02-22T20:16:55.117 INFO:tasks.ceph:Archiving logs... 2026-02-22T20:16:55.118 DEBUG:teuthology.misc:Transferring archived files from trial007:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974/remote/trial007/log 2026-02-22T20:16:55.122 DEBUG:teuthology.orchestra.run.trial007:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-22T20:16:56.288 DEBUG:teuthology.misc:Transferring archived files from trial017:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974/remote/trial017/log 2026-02-22T20:16:56.289 DEBUG:teuthology.orchestra.run.trial017:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-22T20:16:58.677 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-22T20:16:58.726 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-22T20:16:58.726 DEBUG:teuthology.orchestra.run.trial007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T20:16:58.728 DEBUG:teuthology.orchestra.run.trial017:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T20:16:58.871 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-22T20:16:58.882 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-22T20:16:58.883 DEBUG:teuthology.orchestra.run.trial007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T20:16:58.884 DEBUG:teuthology.orchestra.run.trial017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T20:16:58.940 INFO:teuthology.orchestra.run.trial007.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T20:16:58.940 INFO:teuthology.orchestra.run.trial007.stdout:============================================================================== 2026-02-22T20:16:58.940 INFO:teuthology.orchestra.run.trial007.stdout:-172-104-28-175. 82.35.162.146 2 u 18 128 377 3.317 +6.693 0.229 2026-02-22T20:16:58.940 INFO:teuthology.orchestra.run.trial007.stdout:+144.202.41.38 ( 99.28.14.242 2 u 30 128 377 33.690 -0.424 1.841 2026-02-22T20:16:58.940 INFO:teuthology.orchestra.run.trial007.stdout:*jane.qotw.net 192.58.120.8 2 u 90 128 377 41.286 -0.463 1.811 2026-02-22T20:16:58.940 INFO:teuthology.orchestra.run.trial007.stdout:+ntp1.lax.us.hoj 2.148.140.233 2 u 105 128 377 63.780 +1.697 0.746 2026-02-22T20:16:58.949 INFO:teuthology.orchestra.run.trial017.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T20:16:58.949 INFO:teuthology.orchestra.run.trial017.stdout:============================================================================== 2026-02-22T20:16:58.949 INFO:teuthology.orchestra.run.trial017.stdout:+x.ns.gin.ntt.ne 129.250.35.222 2 u 19 64 377 3.026 -6.922 1.031 2026-02-22T20:16:58.949 INFO:teuthology.orchestra.run.trial017.stdout:+as393746.mci.tr 42.20.202.230 2 u 24 64 377 29.821 -4.779 1.409 2026-02-22T20:16:58.949 INFO:teuthology.orchestra.run.trial017.stdout:*138.89.14.60 (p .PPS. 1 u 25 64 367 5.656 -5.716 0.672 2026-02-22T20:16:58.949 INFO:teuthology.orchestra.run.trial017.stdout:-73.185.182.209 204.117.185.216 2 u 8 64 377 44.985 -7.319 2.938 2026-02-22T20:16:58.950 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-22T20:16:58.961 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-22T20:16:58.962 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-22T20:16:58.976 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-22T20:16:58.988 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-22T20:16:59.001 INFO:teuthology.task.internal:Duration was 1933.126708 seconds 2026-02-22T20:16:59.001 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-22T20:16:59.015 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-22T20:16:59.015 DEBUG:teuthology.orchestra.run.trial007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T20:16:59.017 DEBUG:teuthology.orchestra.run.trial017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T20:16:59.042 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-22T20:16:59.042 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial007.front.sepia.ceph.com 2026-02-22T20:16:59.043 DEBUG:teuthology.orchestra.run.trial007:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T20:16:59.086 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial017.front.sepia.ceph.com 2026-02-22T20:16:59.087 DEBUG:teuthology.orchestra.run.trial017:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T20:16:59.095 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-22T20:16:59.095 DEBUG:teuthology.orchestra.run.trial007:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T20:16:59.129 DEBUG:teuthology.orchestra.run.trial017:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T20:16:59.358 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-22T20:16:59.358 DEBUG:teuthology.orchestra.run.trial007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T20:16:59.361 DEBUG:teuthology.orchestra.run.trial017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T20:16:59.367 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T20:16:59.369 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T20:16:59.369 INFO:teuthology.orchestra.run.trial007.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T20:16:59.369 INFO:teuthology.orchestra.run.trial007.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T20:16:59.370 INFO:teuthology.orchestra.run.trial007.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T20:16:59.371 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T20:16:59.371 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T20:16:59.371 INFO:teuthology.orchestra.run.trial017.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T20:16:59.371 INFO:teuthology.orchestra.run.trial017.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T20:16:59.371 INFO:teuthology.orchestra.run.trial017.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T20:16:59.437 INFO:teuthology.orchestra.run.trial007.stderr: 81.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T20:16:59.439 INFO:teuthology.orchestra.run.trial017.stderr: 81.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T20:16:59.441 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-22T20:16:59.460 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-22T20:16:59.460 DEBUG:teuthology.orchestra.run.trial007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T20:16:59.487 DEBUG:teuthology.orchestra.run.trial017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T20:16:59.495 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-22T20:16:59.506 DEBUG:teuthology.orchestra.run.trial007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:16:59.532 DEBUG:teuthology.orchestra.run.trial017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:16:59.539 INFO:teuthology.orchestra.run.trial007.stdout:kernel.core_pattern = core 2026-02-22T20:16:59.549 INFO:teuthology.orchestra.run.trial017.stdout:kernel.core_pattern = core 2026-02-22T20:16:59.557 DEBUG:teuthology.orchestra.run.trial007:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:16:59.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T20:16:59.595 DEBUG:teuthology.orchestra.run.trial017:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:16:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T20:16:59.605 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-22T20:16:59.623 INFO:teuthology.task.internal:Transferring archived files... 2026-02-22T20:16:59.624 DEBUG:teuthology.misc:Transferring archived files from trial007:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974/remote/trial007 2026-02-22T20:16:59.624 DEBUG:teuthology.orchestra.run.trial007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T20:16:59.675 DEBUG:teuthology.misc:Transferring archived files from trial017:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:38-rbd-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/63974/remote/trial017 2026-02-22T20:16:59.676 DEBUG:teuthology.orchestra.run.trial017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T20:16:59.715 INFO:teuthology.task.internal:Removing archive directory... 2026-02-22T20:16:59.715 DEBUG:teuthology.orchestra.run.trial007:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T20:16:59.717 DEBUG:teuthology.orchestra.run.trial017:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T20:16:59.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-22T20:16:59.753 INFO:teuthology.task.internal:Not uploading archives. 2026-02-22T20:16:59.753 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-22T20:16:59.771 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-22T20:16:59.771 DEBUG:teuthology.orchestra.run.trial007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T20:16:59.773 DEBUG:teuthology.orchestra.run.trial017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T20:16:59.776 INFO:teuthology.orchestra.run.trial007.stdout: 1179659 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 20:16 /home/ubuntu/cephtest 2026-02-22T20:16:59.785 INFO:teuthology.orchestra.run.trial017.stdout: 1179687 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 20:16 /home/ubuntu/cephtest 2026-02-22T20:16:59.786 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-22T20:16:59.798 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-22T20:16:59.823 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-journaling} duration: 1933.126707792282 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-22T20:16:59.823 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T20:16:59.933 INFO:teuthology.run:pass